Projects at scale

An artist saves a 64 MB Photoshop file, submits it to version control, then adjusts one sky layer and submits again. The first submit stores 64 MiB on the server. The second stores 2.7 MiB. The other 61.3 MiB never moved, because the server already had those bytes and both sides could prove it.

That is the whole subject of this piece: not the number (64 MB is small by game standards; more later) but the machinery, and what falls out of taking it seriously: partial checkouts of enormous projects, exclusive locks on un-mergeable files, governed sharing across projects, and feeding the same bytes to a floor of workstations without melting the origin.

Game development already has a good answer to most of this, and it isn't new. Perforce solved large binaries decades ago: exclusive checkouts, path-level permissions, proxies at every site, a server that owns the truth. Virtually every studio past a certain size runs it or has; Subversion carried enforced locking into the open-source world too. None of this is news to anyone who has shipped a game. Grixel is built deliberately in that lineage.

The scar tissue lives in two places. Outside the Perforce world (the indie team on a Git host with Git LFS grinding on every clone, the ML lab with a Dropbox side-channel for "the big stuff", the shared drive where a folder named final_v3_FINAL is quietly load-bearing), the workarounds all come from asking a source-code tool to also be a data platform. Inside the Perforce world, the model is right but the substrate is old: each revision of a binary is stored whole, compressed but whole (my sky-layer edit would have cost another full copy), and the metadata layer predates what modern tooling, AI tooling especially, wants to ask a depot.

I'm building Grixel to keep the shape the games industry already trusts and rebuild the storage and metadata underneath it. I have watched both worlds cope for most of my career in games.

#What Grixel is, in one paragraph

Grixel is a server-authoritative version control and data platform for game development and AI/ML work. It is Perforce-shaped, not Git-shaped: one global namespace with paths like /companies/acme/games/seasonal, a server daemon (grixeld) authoritative for metadata, permissions, storage, and atomic changelist commits, and a thin CLI. You don't clone; you check out a view of the subtree you need. The larger ambition is a system of record: the same depot also carries review and sign-off, work items, per-change provenance (including which AI tool produced it), and a governed context feed for AI assistants. That AI layer is the wedge beyond a Perforce reimplementation, but it isn't today's subject. Today is about bytes.

#What content-defined chunking actually does

The core mechanism has an intimidating name and a simple shape. Imagine you archive documents by tearing them into pieces. If you tear at every 100th page (a fixed rule), then inserting three pages into the middle of a manuscript shifts everything after the insertion, and every subsequent piece looks brand new to the archive. If instead you tear wherever a chapter happens to end (a rule derived from the content itself), inserting three pages changes only the one piece containing the insertion. Every other piece still ends at the same chapter boundary and is byte-identical to the copy you already filed. Fingerprint each piece, keep exactly one copy per fingerprint, and storing the edited manuscript costs you one new piece instead of half the archive.

Grixel does this with FastCDC finding chunk boundaries via a rolling hash (a dual profile tuned by file size and extension) and BLAKE3 fingerprinting each chunk. Every file becomes a manifest of chunk hashes, the store keeps one copy of each chunk, and both submit and sync speak in chunks, so a small edit to a big file moves only the new chunks in either direction. The downstream half matters just as much: a teammate syncing the edited key art fetches only the new chunks, and delta-pull assembles the rest from what their machine already has.

· · ·

#Chunks on a plain filesystem

The first design decision is where those chunks live: a flat, content-addressed layout on the server's local filesystem. Not an object store.

The reflexive modern answer is S3 or something S3-shaped, and it buys real things: durability you don't have to think about, capacity without ceilings. The price is an indirection between you and every read. Each chunk fetch becomes a remote call through another service's latency and failure modes, and the deployment stops being "one binary and a disk." For the workload this system is aimed at (a studio or lab LAN, many workstations hammering one authoritative server all day) I want reads to be a filesystem operation and the whole thing runnable by a single operator on commodity hardware. Backup is deliberately boring: archive the data root.

Two operational notes. The backup tooling takes an exclusive lock on the data root, so today a backup is a stop-the-server or filesystem-snapshot operation; the compensating win is chunk immutability, which makes incremental backup cheap (only new chunks appear between runs) and lets grixeld verify re-hash every chunk against its stored name, so the store proves its own bits. History need not grow forever: unreachable objects are garbage-collected, and grixel ref rm retires an abandoned ref, with scope.lock-pinned commits protected as GC roots so cleanup cannot break a reproducible mount.

The trade in return: durability is the operator's problem rather than a cloud provider's promise, and one machine's disks have a ceiling. The scale-out answer is a caching read proxy in front of the store (the flat content-addressed layout makes it almost trivial); more on that below.

A note on the commands: these are the real grixel/grixeld binaries in local mode against a scratch data root, hence -as alice asserting a principal directly; in a team deployment you grixel login once and the flag disappears, identity coming from the server-verified token. Output is verbatim; workspace paths are shortened to ~/demo.

The file:

$ ls -lh keyart_main.psd
-rw-r--r--@ 1 joeblogs  wheel    64M  6 Jul 00:32 keyart_main.psd

The first submit, all new:

$ grixel submit -as alice -ref companies/acme/games/seasonal -at art -m "main key art" keyart_main.psd
grixel: holding exclusive lock on /companies/acme/games/seasonal/art/keyart_main.psd (lockable type)
Committed: 2565dde94e79baae2337885ed2cbd60bc022633e3b359eefb1dd895dcae39b8c
  ref     : companies/acme/games/seasonal
  changes : 1 files, 64.0 MiB
  stored  : 64.0 MiB new, 0 B deduped
  parent  : 4ee43c8d07ae3d068dccfa44bc074e0a309b073183536f647dd365cb4f61ae50
  author  : alice
  message : main key art

(Ignore the exclusive-lock line; it gets its own section below.)

Then the sky-layer edit, saved and submitted again:

$ grixel submit -as alice -ref companies/acme/games/seasonal -at art -m "key art: adjust sky layer" keyart_main.psd
grixel: holding exclusive lock on /companies/acme/games/seasonal/art/keyart_main.psd (lockable type)
Committed: a633634ee24180a7bca0559d152888292cb9a5432c7e4ae63b1e78067f554e59
  ref     : companies/acme/games/seasonal
  changes : 1 files, 64.0 MiB
  stored  : 2.7 MiB new, 61.3 MiB deduped
  parent  : 2565dde94e79baae2337885ed2cbd60bc022633e3b359eefb1dd895dcae39b8c
  author  : alice
  message : key art: adjust sky layer

The file is still 64.0 MiB; the commit stored 2.7 MiB. Photoshop rewrote parts of the file on save, but rolling-hash boundaries left most chunks byte-identical to ones the server already held, so they were never re-sent. Inspect the carving:

$ grixel stat -as alice companies/acme/games/seasonal art/keyart_main.psd
/companies/acme/games/seasonal/art/keyart_main.psd
  size    : 64.0 MiB (67108864 bytes)
  chunks  : 15
  profile : large-asset (target chunk ~4.0 MiB)
  mode    : 0100644
  hash    : a724bb3cc5c0fb0938548fc30ca4e55a8e118aa77d5ab7efa47ebdd58f2f7f3e
  (chunks are the content-addressed dedup unit; cross-version sharing is reported at submit)

In plain terms: the file is stored as fifteen pieces, and history only pays for the pieces that changed. The pieces average about 4.3 MiB under the large-asset profile, which is also the honest granularity limit: the chunk is the unit of dedup, so a one-byte edit still stores one whole new chunk, roughly 4 MiB here. Ten revisions cost the first revision plus the changed chunks each time, and the same arithmetic runs downstream on sync.

Dedup is also store-wide by content; the cheapest demonstration is a copy. Marketing wants the same key art at a second path:

$ grixel submit -as alice -m "storefront wants the same key art"
grixel: holding exclusive lock on /companies/acme/games/seasonal/art/keyart_storefront.psd (lockable type)
Committed: 5843571f184131af97c8077d90a052af2d9d984ee5994b3f08989d5675b4948f
  ref     : companies/acme/games/seasonal
  changes : 1
  stored  : 0 B new, 64.0 MiB deduped
  parent  : f64417def5ed20f7203545e7b98e7920a9fccfea39c2db08744e65dd82c42c0b
  base    : 5843571f184131af97c8077d90a052af2d9d984ee5994b3f08989d5675b4948f (advanced)

Zero new bytes; all 64 MiB already lived in the store under another name. (Format notes, once: workspace submits print a leaner summary than one-shot submits, a bare changes : 1 with no byte count; the base : ... (advanced) line is sync bookkeeping, the workspace's base advancing to its own new commit.)

The contrast with the incumbent deserves precision, because Perforce gets so much else right. Helix Core stores text revisions as deltas but stores each binary revision whole: compressed by default, whole nonetheless. The model is sound; the storage layer simply predates content-defined chunking. Chunk-level dedup on binaries is the single biggest thing Grixel's store adds, and it lands on exactly the files that dominate a game project's bytes. One caveat in Helix's favor: branching is cheap there through lazy copies. The whole-revision cost is paid per edited revision, and for iterated art that is exactly the expensive case.

#Take only what you need

The next decision is about the working copy. If a project ref holds hundreds of gigabytes of art, a gameplay programmer should not have to materialize any of it to edit a spawn table.

The seductive answer is a virtual filesystem: mount the depot as a drive and files materialize on open. Perforce-world studios use virtualized workspaces, and it is genuinely pleasant when it works. I chose not to build it for v1: it means a kernel-level driver on three operating systems and a good answer to an ugly question (what happens to an application holding a half-materialized file when the server goes unreachable?). FUSE-style mounts are on the roadmap; they are not where I wanted the correctness budget spent first.

Modern Git deserves a named mention: git sparse-checkout with partial clone genuinely reduces what lands on disk, and for one big repository of code it works. What it cannot give this workload is a server-enforced view over a shared namespace (it is per-repository, and the sparseness is client-side convention rather than server policy), and the heavy binaries still route through Git LFS's pointer seam.

The v1 answer is explicit sparse checkout. You declare the paths you want, or the ones you don't:

$ grixel checkout -as bob -exclude 'art/**' companies/acme/games/seasonal sparse-ws
Checked out companies/acme/games/seasonal
  dir   : ~/demo/sparse-ws
  base  : 3c4bf58d08b850204e2140fa0a8fb1721468b5917536da5c9278280bd3639fcb
  files : 2
  view  : all (excl. art/**) (partial checkout)

Edit files, then: grixel changes / grixel diff / grixel submit -m ...

And what actually landed on disk:

$ du -sh sparse-ws
 20K    ~/demo/sparse-ws

Twenty kilobytes. The key art exists in the same ref, in the same history, governed by the same permissions, and this workspace simply never pulled it. grixel sync can widen or narrow the view later, and the include form is symmetrical: grixel checkout -path 'design/**' materializes only the named subtree, which is how a build node pulls one folder out of a huge ref. The cost: no virtual drive, no files appearing on open; you say what you want before you get it. For v1, I'll take the predictability.

#There is no merging a Photoshop file

Text merges. Binaries don't. When two people edit the same .psd, there are no conflict markers to reconcile; someone's work gets discarded; the only question is whether the tooling admits that up front or shrugs after the collision.

Perforce got this right decades ago with exclusive checkout (Subversion has an equivalent in its needs-lock property), and it is one of the concrete reasons studios stayed on Perforce while the rest of the software world moved to Git. Git's modern answer is git lfs lock: server-backed, enforced at push on the big forges, for files that are LFS-tracked with locking switched on. Real, but clunky and partial; it guards the pointer layer of one repository, and plenty of teams never enable it. A lock is a promise about a file's future, and promises want a single authority; a server-authoritative design gets locks almost for free, enforced at the one place no client can route around: the submit chokepoint.

Grixel adds a second layer: an admin declares whole file types un-mergeable, making a lock a requirement to submit rather than optional politeness.

$ cat data/.grixel/lockable.yaml
version: 1
extensions: [".psd", ".fbx", ".uasset", ".umap", ".wav"]

The discipline that actually saves an afternoon is claiming the lock before you open the file, the same muscle memory as p4 edit. Alice claims the boss arena first thing:

$ grixel edit -as alice art/boss_arena.uasset
editing /companies/acme/games/seasonal/art/boss_arena.uasset — exclusive lock held (required for this type)

When Maya reaches for the same file at nine in the morning, the collision surfaces before she has painted a stroke:

$ grixel lock -as maya /companies/acme/games/seasonal/art/boss_arena.uasset
grixel: lock /companies/acme/games/seasonal/art/boss_arena.uasset: locked by alice — run 'grixel locks /companies/acme/games/seasonal/art/boss_arena.uasset' for details

That refusal costs Maya ten seconds. Alice finishes and submits ("boss arena: relight the pit" lands as 19.8 MiB new, 28.2 MiB deduped on a 48 MB asset; the dedup line rides on every submit), the lock releases with the commit, and Maya can claim it the moment it is free:

$ grixel lock -as maya /companies/acme/games/seasonal/art/boss_arena.uasset   # after the release
locked /companies/acme/games/seasonal/art/boss_arena.uasset (holder maya, id 6c43eb2a53ec5aeb)

Now the version where the discipline slips. Alice holds the lock on the key art (claimed with the same grixel edit); Bob, who has claimed nothing, edits his local copy anyway. Nothing stops him; it's his disk. The stop comes at submit, and it is loud:

$ grixel submit -as bob -m "bob tweaks key art"
grixel: /companies/acme/games/seasonal/art/keyart_main.psd is a lockable type and its exclusive lock couldn't be claimed: repo: path is locked by another user: /companies/acme/games/seasonal/art/keyart_main.psd held by alice
  (someone else may hold it — see 'grixel locks'; you can't submit it until it's free)

$ grixel revert -as bob art/keyart_main.psd
reverted 1 path(s)

Notice what actually happened. Bob did lose work: whatever he painted, he reverted away. The lock did not save his afternoon; claiming first would have. What it guarantees is the floor: the collision is refused loudly at the chokepoint instead of Bob silently overwriting Alice, and because .psd is a lockable type, the floor holds even for people who never think about locks. That is also the exclusive-lock line in the very first transcript: Alice was covered before anyone configured anything per-file.

One ergonomic gap against Perforce is worth naming. Perforce nudges the claim-first habit physically: working files sit read-only until checkout, so the moment Photoshop refuses to save, an artist knows to claim the file. Grixel's ordinary working files are writable (mapped and vendored files materialize read-only; regular project files do not), so nothing in the filesystem pushes you toward grixel edit yet. The enforcement floor is the same; the habit-forming friction is missing.

Locks serialize work on a file, which is exactly their job, and they sting when someone leaves for a long weekend holding one. There is an admin break-glass release for that case, but the coordination cost doesn't vanish; it becomes visible instead of becoming lost work.

#One namespace, and sharing with a paper trail

Everything so far could exist in a per-project world. Grixel decides otherwise: projects shouldn't be islands.

In a per-repository world, sharing a brand guide or a coding standard across twenty projects means git submodules (pinned, in fairness, but famously sticky: rolling a new pin out to every consumer is its own chore), hand-vendored copies, or a wiki nobody trusts. In one global namespace, a project can declare exactly what it pulls from elsewhere, in a committed file:

$ cat .grixel/scope.json
{
  "name": "SeasonalEvent2026",
  "owner": "team:seasonal",
  "description": "Spring festival content drop",
  "mappings": [
    {
      "source": "/companies/acme/standards/brand",
      "mountAt": ".standards/brand",
      "pin": { "mode": "head" }
    },
    {
      "source": "/companies/acme/standards/coding/go",
      "mountAt": ".standards/go",
      "pin": { "mode": "head" }
    }
  ],
  "ai_hints": "Tone: playful spring festival. Palette and voice live in .standards/brand."
}

Running grixel scope update resolves each mapping to a concrete revision and commits the pins as a scope.lock, so the pins themselves are history.

The part I care about most is when the upstream moves. The brand team commits a new revision of the voice guide, and ordinary status output flags it:

$ grixel status -as alice   # after the upstream edit
ref      : companies/acme/games/seasonal
base     : 9d42cd83 (up to date)
changes  : none (clean)
mappings : 2 total, 1 current, 1 drift-on-head
  ↻ .standards/brand (drift-on-head)
  → run 'grixel scope update' to refresh the mappings

Staleness is a first-class, visible state; you find out from grixel status rather than from a bug report. Adopting the new upstream is a deliberate grixel scope update that lands as a reviewable commit to scope.lock. Reproducibility becomes structural: any revision of the project checks out with the exact standards it was built against. Mappings are read-only in v1, deliberately; a project consumes shared subtrees, it doesn't write back through them.

#Getting the same bytes to forty machines

The classic studio morning: an asset drop lands overnight, forty workstations sync at 9am, and the origin (or the WAN link to it) saturates. Perforce's answer for decades has been the p4p proxy, a cache at each site so the origin serves each file once per office instead of once per person.

Grixel's equivalent is a caching read proxy in front of the chunk store, where the flat, content-addressed layout pays a second dividend: a chunk's name is its BLAKE3 hash and the hash proves the content, so the proxy needs no knowledge of files, projects, or permissions above it. Cache hit: serve from local disk on the LAN. Miss: fetch from origin once, keep it, serve everyone else. Dedup compounds with caching: the proxy stores one copy of a chunk however many files share it, so fifteen artists syncing fifteen slightly different key-art revisions draw mostly the same chunks from the same nearby cache.

The proxy is implemented and merged, and it fronts reads for authenticated clients; its multi-node behavior, cache sizing included, is exactly what the unpublished benchmark has to show: multi-gigabyte assets, many client nodes, measured origin offload and wall-clock sync times. The number is unmeasured in public, so I am not going to state one.

· · ·

#What this does not prove yet

Here is the honest ledger, the part a skeptic should quote.

The demo file is 64 MB. Real depots run to hundreds of gigabytes per project and terabytes overall, with individual files far larger. The mechanisms shown here are size-independent by construction, and the large-asset profile exists precisely for big files, but "by construction" is not the same as "demonstrated at 400 GB," and I have not published numbers at that scale.

File count is a separate axis from byte count, also unmeasured: I have not published the cost of status or sync against a million-file view.

The caching read proxy has no published multi-node benchmark. The design mirrors a topology Perforce validated over decades, which is a reason for confidence, not proof.

There are no FUSE virtual mounts. Artists and build machines must check out, sparsely or fully; nobody browses a virtual drive today. Teams whose workflow assumes virtualized workspaces will feel that immediately.

There is no multi-region replication. One origin owns writes; proxies are a read-side answer. A studio with heavy write traffic on two continents is not served well yet.

Commits serialize through a single writer per data root: a deliberate simplicity-and-correctness choice (the pipeline is write-ahead-logged, crash-recoverable, and fault-injection tested). At some commit rate that becomes the constraint, and I have not measured where.

There are no Unreal or Unity plugins yet; engine-side integration is manual today, via the CLI, the Go and Python SDKs, or the versioned --json output.

And the product is pre-release, full stop: real, running code, none of it with years of production scar tissue behind it yet.

#The idea worth keeping

All the workarounds I opened with share one root cause: the moment a file gets big, it stops being fully in version control. Git LFS splits every tracked file into a pointer in history and bytes in a separate store, with its own quota and failure domain and no chunk-level dedup between revisions. The Dropbox folder has no history worth the name. final_v3_FINAL is a coping mechanism wearing a filename. Perforce is the standing counterexample, and the studios with the biggest files in the industry have been quietly right about its shape for decades: give the data a server-authoritative home and the workarounds evaporate.

A better attachment mechanism bolted onto a source-code tool will not close that gap. Treating large binary data as the primary workload will: chunk it so history is cheap, check out slices so scale doesn't tax everyone, lock what cannot merge so collisions surface before the painting starts, share it across projects with pinned, visible freshness, and cache it near the people who need it. Once the heavy data lives under the same roof, history, and permissions as everything else, the depot can honestly claim to be the database of record for the whole operation.

That is what Grixel is: the shape the games industry already proved, with the store and the metadata rebuilt for this decade's data.

Grixel is pre-release. Read the docs, or request early access.