CFS

CFS is a virtual file system for video games and game engines. It packs multiple files inside bundles and navigate through the system with an index file. It reduces the number of files. CFS allows you to divide files to place different parts in different bundles.

CFS se decompose en deux parties:

• Containers (aka bundles) are the files that contain parts of files.
• The index which contains all the information (filename, id, size, chunks). We can extract the files from the containers with the index. Here is an example of an index in JSON format:

  
  {
      "root": "",
      "containers": [
          {
              "guid": "b4248b9e-211d-11ea-97a2-4ccc6a46dfb9",
              "path": "b4248b9e-211d-11ea-97a2-4ccc6a46dfb9.pak"
          },
          {
              "guid": "d092a0cb-211d-11ea-9eaf-4ccc6a46dfb9",
              "path": "d092a0cb-211d-11ea-9eaf-4ccc6a46dfb9.pak"
          }
      ],
      "entries": [
          {
              "guid": "b4248b9f-211d-11ea-8988-4ccc6a46dfb9",
              "path": "img1.png",
              "blocks": [
                  {
                      "container": "b4248b9e-211d-11ea-97a2-4ccc6a46dfb9",
                      "offset": 0,
                      "size": 53373
                  }
              ],
              "hash": {
                  "md5": "e5b36cb5264a7ce5a2d33e14d587d1cc",
                  "sha1": "15237cfd9787f60095ae242c3201766e1b0e29ec"
              }
          },
          {
              "guid": "b424b2cf-211d-11ea-85cb-4ccc6a46dfb9",
              "path": "img2.png",
              "blocks": [
                  {
                      "container": "b4248b9e-211d-11ea-97a2-4ccc6a46dfb9",
                      "offset": 53373,
                      "size": 63236
                  },
                  {
                      "container": "d092a0cb-211d-11ea-9eaf-4ccc6a46dfb9",
                      "offset": 0,
                      "size": 69851
                  }
              ],
              "hash": {
                  "md5": "cdc7dbc4baf0cf17f7ddc6c7a108d992",
                  "sha1": "d41e28bc9db2238bad0d333399f7732cc4a50324"
              }
          }
      ]
  }
                  

CFS has several advantages:

• Easy to implement
• Check file integrity with SHA1 and MD5 algorithms (This feature can be disabled to improve performance)
• The index uses a strict JSON structure (without dynamic objects) that can be loaded very easily by serializers (like JsonUtility from Unity).
• The index can be encrypted to prevent opening, viewing and decompressing files, at minimal runtime cost.
• Open source (MIT License)

I designed and implemented the CFS because of a limitation of the number of files with Android systems. While working on a project, I realized that the more files there were in a folder on Android, the slower it was to read and write the files (About 10 min to transfer a 50Ko file in a folder with already 3500 images). With the CFS, I managed to group 40000 textures (about 1.5Go) in 8 files of 200Mo each.

Implementations

Unity

CFS has a Unity implementation. This implementation supports both JSON format and binary format. Sources

Packer

Packer is a tool written in python to create the index and the containers from a set of files. It is a naive implementation designed to arbitrarily group files together. Sources