One Storage Model: The Bucket Is the Disk
Most platforms have a machine disk and a network volume, two models with different failure modes. Pilots has one: the root is a read-through cache over object storage, so wiping a host's NVMe costs a download and never a rebuild.
Ask what happens to your data when a host dies and most platforms have two answers, because they have two storage models.
There is the machine's own disk, which is a copy on that host's drive. That is fast and it is why the machine is pinned there. Moving it means copying it, which is why migration is a feature with an asterisk.
Then there is a network volume you attach for the data you actually care about, which survives independently and has its own performance characteristics, its own attach semantics, and its own failure modes.
Two models is not a design error. It is what you get by solving the fast case first and the durable case second, and both solutions are correct. It just means every question a user asks has two answers, and the platform has two sets of edge cases to get right for ever.
Pilots has one model. The root disk and the volume are the same S3-backed thing with different durability promises, and the host's NVMe is a cache of it.
The design test is one sentence
Wipe any host's disk and nothing is lost.
That is the whole invariant, and it is worth stating as a test rather than as an aspiration because it is checkable. Every time I have been tempted to keep something important on local disk for performance, that sentence has been the thing that stopped me, and every time the cached version turned out to be fast enough.
What actually lives on the host is a cache directory of content-addressed builds and, per machine, a sparse overlay file holding the blocks that machine has written. Both are reconstructible from the bucket. Neither is authoritative.
Read-through, not copy-on-create
The naive way to do "the disk is in object storage" is to download it before you start the machine. That makes create slow in proportion to the image size, which is the thing this engine exists not to do.
Instead the block handler opens the template by whichever of its two names is usable.
If the local build directory has a data.complete marker, it reads from local disk. If it does not, it serves the template from the bucket, range by range, caching each range into that directory as it goes, while a background prefault pulls the rest. So a host that has never held the template attaches the device immediately and reads do not block on hydration. The machine starts now, and the host gets faster over the next few seconds.
The marker is the interesting bit. It is a marker rather than a size check, because an interrupted pull leaves a full-length file of holes. A size check passes on that file and you read zeros where the template had content, which is the failure mode a sparse file hands you in a different disguise. A build that is only partly here is also refused as a chunkify parent rather than quietly diffed against.
On top of the template sits the machine's own overlay: a sparse memory-mapped file plus a bitmap of dirty 4 KiB blocks. A read hits the overlay only if every block it covers is dirty, otherwise it falls through to the template. A machine that has changed nothing stores nothing, and no per-machine copy of a root filesystem exists on any path in the system.
Two promises, both published
One model does not mean one guarantee. The two tiers are stated rather than implied, because a storage system that is vague about durability is worse than one that is slow.
A volume write is durable when it returns. Per write, no buffering, and --writeback is deliberately absent from the mount.
A machine root write is durable as of the last checkpoint, suspend, or periodic root flush, whichever is most recent. The window defaults to 60 seconds, is operator-tunable, and is measured rather than asserted: pilots_root_flush_lag_seconds says how far the bucket trails the disk and pilots_root_flush_pause_seconds says what the flush costs the guest.
The sentence that follows from those two, which is in the docs in these words: data that cannot afford to lose its last 60 seconds belongs on a volume. That is a real limit, written where somebody deciding will read it, rather than a footnote under a durability claim.
The flush and what it cost has its own post, including the axis where we are behind the comparable product and why.
Why not pin the disk to the host
Fly has written publicly about what host-attached storage costs to operate, and their own account is that it "took 3 years to get workload migration right with attached storage, and it's still not 'easy'". I find that quote persuasive in a way a benchmark never is. It is a team with far more operational experience than I have saying the model is hard after three years of working on it.
The shape worth matching is the one Sprites uses, where the whole root is a read-through cache over object storage. The shape worth avoiding is three storage models, which is where e2b ended up, with volumes that only exist on one cloud.
So the machine root here is S3-backed all the way down. The truth of a root is its chunked build chain in the bucket. The host's build directory and the machine's overlay are caches of that truth, disposable and rebuilt on demand.
What it gets you, concretely
A machine whose host dies comes back on a survivor, from object storage, with the same URL, its disk intact as of the published window, and no human involved. That works because the survivor needs nothing from the dead host. Not a copy, not a handoff, not a volume detach that has to succeed first.
A machine can be woken on a host that has never run it, which is what makes suspend genuinely free: there is no reason to keep anything warm on the host that last ran it, because that host has no special relationship to the machine.
And the operational story for a full or failing disk is "wipe it". The host re-hydrates what it needs and keeps serving. I have done this on a live host while it was serving traffic, which is not a sentence I could have written about the storage model this replaced.
Every note, or install the CLI and try the thing this one is about.