What Pilots Is: One Primitive for Sandboxes and Production
Pilots runs Firecracker microVMs as both instant sandboxes for AI agents and durable production services, on one primitive, with no central control plane. The origin story and the three decisions everything else follows from.
I had two problems that looked like one problem, and every product I tried solved exactly one of them.
The first was agents. An AI coding agent that can only write code is half useful. It needs somewhere to run what it wrote, somewhere that can be broken and thrown away, somewhere that exists before the thought finishes rather than ninety seconds later. The sandbox products do this well.
The second was the thing that happens next. The agent writes something that works, I look at it, and I want it to keep running with a URL people can visit. That is a deploy product. So the prototype gets rebuilt somewhere else, on a different platform, with a different CLI, a different config file, and a new address. The state it accumulated while I was working on it does not come along.
Both halves are solved. They are just solved by different products, and the seam between them is where all the work is. Fly and Sprites are the clearest version of this, and I mean that as a compliment to both: they are two good products built by one company, and you still install two CLIs (fly and sprite) to use them. The sandbox and the service are separate things that were combined after the fact.
Pilots is what you get if you refuse the seam from the start. One primitive, two faces, one CLI.
One primitive means the lifecycle is config, not a product tier
A machine on Pilots is a Firecracker microVM with an identity that never changes. A sandbox is one of those. A production service is one of those. They are not two types with a conversion path between them, because there is nothing to convert.
What differs is three knobs on the row:
autoStopisoff,stoporsuspend. A sandbox suspends when nobody is using it. A service under load does not.autoStartdecides whether a request to a sleeping machine wakes it, which is how scale-to-zero works without anyone watching.minMachinesRunningis the floor. Zero is the default on both faces, which is to say the default is that you pay for nothing when nothing is happening.
pilot promote moves a sandbox to the production face. It keeps the URL, the disk, the checkpoints, the agent token and the machine id. What it adds is a release to roll back to, a health gate on every deploy, replicas that come up under load, and a domain of its own. Nothing is rebuilt, because there is no other shape to rebuild into. The machine you were poking at is the machine serving traffic.
That is the whole product thesis, and every design decision below is downstream of it.
No control plane, and I mean none
Every host in the fleet runs the same three processes. hostd is ours and is the entire data plane, corrosion is Fly's gossip-replicated SQLite, and firecracker is spawned per machine. That is the list. There is no scheduler tier, no managed database, no load balancer appliance, and no API server that the other hosts are clients of.
Every host serves the full API. A create, a deploy, an exec, a DNS answer and a wake are all answered by whichever host the request happened to reach, out of that host's own local replica of the state. A lookup is a read from a SQLite file on local NVMe, in microseconds, with no network call in the request path at all.
The shape is borrowed from uncloud, which is Apache-2.0 and worth reading. What it buys is a failure mode that does not exist: there is no machine whose death takes the API down, because there is no machine the API runs on. Adding a host is scripts/host-bootstrap.sh <ip>, it joins the gossip mesh, and it starts taking traffic. Nothing registers it with anything.
The cost is real and I will not pretend otherwise. Gossiped CRDT state is last-write-wins, which means no uniqueness constraints and no cross-host transactions, which means the correctness burden moves from the database into the code that writes rows. The rule that holds it together is that a host writes only rows describing its own machines. Violating that does not produce an error. It produces a silent merge that corrupts state later, somewhere else, in a way that is very hard to trace back. That invariant is enforced in review, and it is the one I am strictest about.
Object storage is the disk, and the host disk is a cache
A machine's root filesystem lives in S3-compatible object storage, content-addressed and chunked. The host's NVMe holds a cache of it. The design test I hold this to is one sentence: wipe any host's disk and nothing is lost.
Most platforms have two storage models. There is the machine's own disk, which is a local copy that has to be moved when the machine moves, and there is a network volume you attach for the data you actually care about. Fly has been honest about what the first one costs. Their own words are that it "took 3 years to get workload migration right with attached storage, and it's still not 'easy'". That is a company with far more operational practice than I have saying the model is hard, which is a good reason to not choose the model.
So there is one model here. The root disk and the volume are the same S3-backed thing with different durability promises, and the promises are published rather than implied. A volume write is durable when it returns. A root write is durable as of the last checkpoint, suspend or periodic flush, whichever is most recent, with a window that defaults to sixty seconds and is exposed as a metric rather than as a claim. Data that cannot afford to lose its last sixty seconds belongs on a volume, and the docs say so in those words.
The consequence that matters day to day is that machines are not attached to hosts. A machine whose host dies comes back on a survivor, from object storage, with the same URL. Nobody pages anybody.
Nothing boots, because booting is where the second went
A Linux guest takes as long to boot as the operating system takes, every time, which is why sandbox products either make you wait or keep idle virtual machines burning money. Pilots does neither. Each host boots one golden template once, lets systemd settle for about twenty seconds, and chunkifies the memory. Every machine after that is a restore of that image with lazy memory behind it, and the pages fault in as the guest touches them.
The budgets on dedicated hardware are create under 500 ms, wake under 200 ms, and a checkpoint resume gap under 500 ms. Those are asserted by the end-to-end battery under PILOTS_E2E_METAL=1, which is the operator asserting real hardware rather than a laptop, because a nested-virtualisation laptop node cannot create a machine in 500 ms whatever else is true about it.
A suspended machine is the other half of the same idea. Suspend kills the Firecracker process, frees the memory, releases the network slot and leaves nothing running. What a suspended machine costs is the storage its snapshot occupies, and that is all. The wake is a held request: a visitor's connection is held open while the machine restores, and then proxied. Not a loading page, not a retry, not a 503 with a "warming up" body.
What I actually use it for
The dashboard at pilots.run is a Pilots service. The marketing site you are reading this on is a Pilots service. The builds that produced them ran inside Firecracker microVMs on the same fleet, because a RUN step in a Dockerfile is arbitrary customer code and does not belong in a container sharing the host kernel with other tenants.
The rest of these notes are the parts that were hard. The order the seven steps of a snapshot have to happen in, and why every one of them was arrived at by measuring a resume gap. Why a copy-on-write file cannot be chunked without its dirty bitmap, and what the bug looks like when you try. The signal that killed one restore in four and reported it as a sizing timeout on a device that looked free. How a fleet decides where a machine goes without anyone electing a leader.
They are notes rather than documentation. If you want the design as a whole, ARCHITECTURE.md in the repo is the source of truth and says so at the top.
Every note, or install the CLI and try the thing this one is about.