Atlas Game Distribution System

Atlas is an application distribution system built primarly for games: it handles publishing new builds and delivering them to players as updates. It is written entirely in C#, using .NET Core and ASP.NET Core.

The project is heavily inspired by the new League of Legends patcher developed by Riot Games, described in the article Supercharging Data Delivery: The New League Patcher. Riot replaced its previous binary-delta system with an approach based on content-defined chunking, which made patching much faster. Atlas applies the same core ideas in a standalone, reusable system.

Its purpose is to deliver updates as efficiently as possible, for both the people shipping them and the players receiving them. To achieve this, Atlas is built around a few key requirements:

• Fast publishing: shipping a new version should take seconds, not minutes.
• Efficient downloading: updates should download quickly, consume as little bandwidth as possible, and keep a small memory footprint on the player's machine.

How can updates be delivered?

There are three common approaches, each with its own trade-offs.

Full archives

The simplest option is to ship an installer or archive containing the whole program and overwrite the existing files. It is easy to set up, but it has a major drawback: players must download the entire game again, even when only a tiny part of it has changed.

Binary deltas

A more refined option is to compute binary deltas (patches) between versions. These patches can be extremely small, which makes updates quick to download, but generating them is complex and very slow.

The approach also scales poorly. Each delta is computed from two complete versions of the game, so every new release requires several patches: one from each older version still in use. Players returning after a long break either need a dedicated patch or have to apply a long chain of them, and the processing time on the publishing side keeps growing with every release.

File chunking

This is the approach Atlas uses, and it offers a strong middle ground between the two others:

• Only the version being released is needed. There is no need to keep or compare previous builds, which makes publishing much faster.
• Only the changed data is uploaded and downloaded. Updates are much lighter than full archives, and only slightly larger than binary deltas.

How does Atlas work?

1. Chunking

Atlas splits files using FastCDC, a content-defined chunking algorithm built on a very fast rolling hash.

Instead of cutting files at fixed intervals, content-defined chunking chooses chunk boundaries based on the data itself, producing chunks of variable size. Each chunk is identified by its content, so updating a file comes down to a simple comparison: the patcher looks at the chunks the new version requires, downloads only the ones it doesn't already have, then rebuilds the file by writing every chunk in the right order, reusing the data already present on disk.

The key advantage is stability: when data is inserted or removed inside a file, most chunks stay exactly the same. With fixed-size chunks (say, every 64 KB), inserting a single byte at the start of a file would shift every boundary after it and change every chunk, turning a tiny edit into a massive update.

This gives chunking a decisive edge over binary deltas: any old version of the game can be brought up to date efficiently, in a single step.

2. Compression

To cut download times further, every chunk is compressed with Zstandard (zstd). Some algorithms achieve slightly better compression ratios, but at the cost of much heavier CPU usage during decompression. Zstd offers very good compression ratios while keeping decompression extremely fast, which makes it the best compromise for this use case.

3. Bundling

Compressed chunks are then grouped into bundles of 60 chunks. Storing each chunk as a separate file would force players to send a huge number of HTTP requests, each fetching only a tiny amount of data. The per-request overhead would slow updates down and could overload the servers. Bundling keeps the number of requests low and downloads efficient.

4. Release manifest

Once bundling is done, Atlas writes the release manifest: a description of every file, chunk and bundle that makes up the release, and of how they fit together.

The manifest was originally stored as JSON, but Atlas now uses FlatBuffers, a binary format that is very compact, loads almost instantly and keeps memory usage low on the client. The manifest itself is also compressed with Zstandard.

5. Downloading and patching

The patcher's role is to take the game as it is currently installed and bring it to the exact state described by the manifest. Depending on the situation, that can mean downloading new chunks, rewriting existing files, removing obsolete ones, and so on.

To know what is already installed, the Atlas patcher keeps a lightweight SQLite database listing every local file and chunk. After downloading the manifest, it compares both and builds an update plan that separates:

• missing chunks, which have to be downloaded;
• chunks already on disk, which are simply copied or moved to where the new files need them.

Missing chunks are fetched with multipart HTTP range requests. When every chunk of a bundle is needed, the patcher downloads the whole bundle; when only some of them are, it requests just the matching byte ranges. Grouping several ranges into a single request keeps the number of requests to a minimum, which is why serving bundles through a CDN that supports range requests is so important.

Results

On a small Unreal Engine project weighing around 500 MB, tested on a machine with an Intel Core i9-9900K (8 cores / 16 threads) and a SATA SSD, the results are excellent:

• the first version deploys in under 30 seconds;
• a second version, 95% identical to the first, deploys in under 4 seconds, since only the modified data needs to be processed.

Current limitations

One of the main issues still to be solved is the handling of very large files. When file sizes are poorly balanced (for example, thousands of small files alongside a single file of several tens of gigabytes), the analysis and chunking phase takes considerably longer.

The reason lies in how chunking works. Although Atlas is heavily multithreaded, content-defined chunking reads a file sequentially: each boundary depends on the data that comes before it, so a single file can only be processed by one thread. The small files are handled quickly in parallel, then every other thread sits idle while one thread works through the huge file on its own. In the end, the largest file, not the total size of the build, dictates how long publishing takes.

A promising solution is to split very large files into big segments (for example, 1 GB each) and chunk every segment in parallel on its own thread. The only cost is a few forced chunk boundaries at the junctions between segments, which slightly reduces deduplication around those points. Since the segments are huge compared to the chunks, this loss is negligible, while the chunking time of large files could drop roughly in proportion to the number of available cores.

Performance also depends heavily on the type of storage. Once chunking runs in parallel, the bottleneck often shifts from the CPU to disk read speed. A fast NVMe SSD can feed many threads at once, a SATA SSD sits somewhere in between, and a traditional hard drive struggles with several simultaneous reads and quickly becomes the limiting factor. The results shown above were measured on a SATA SSD, so an NVMe drive could improve them even further.

What's next

Once finished, Atlas will be open source and freely available on GitHub under the MIT license. It is not intended for production use: it is meant to serve as a foundation for custom distribution systems. It will ship with instructions to run a complete demo using Docker and Kubernetes.