What the hard parts actually cost. Snapshot ordering, lazy memory over userfaultfd, a root disk served from object storage, and a fleet that runs with nothing in the middle.
An AI agent about to run a migration, an upgrade, or a command it just invented is one step from a broken machine. A checkpoint on Pilots restores in place, keeping the same URL and the same agent token, so the undo costs nothing to use.
Scheduled jobs usually need something kept warm to run them. On Pilots a cron fires through the same held-wake path a visitor's request takes, so a scale-to-zero app runs its nightly job with nothing running in between.
Snapshot portability requires every Pilots guest to see the identical network address. Service discovery then has to happen entirely outside the guest, with addresses derived from a host key and translation in the namespace rather than the root.
When every row replicates to every host, a secret in a row is a secret on every disk in the fleet. How Pilots resolves secret references client-side, seals what it stores, and what that defends against.
A credential baked into a guest is in every snapshot and every fork of that guest. On Pilots a machine asks for a token over a socket that only it can reach, and the request path is the identity.
Suspend, wake, checkpoint, restore, promote, redeploy and host death all leave a Pilots machine's address alone. What it takes to mean that, from one wildcard certificate to who is allowed to fetch a URL at all.
If the client decides whether a successful build becomes a release, then a closed lid is a build that deployed nothing. On Pilots the deploy intent travels with the build and the host that ran it cuts the release.
Pilots detects compose files, Dockerfiles and known frameworks, and when it recognises nothing it returns a 400 carrying everything an agent needs to write a Dockerfile itself. Why there is no buildpack behind that.
Taking a flattened filesystem out of BuildKit means throwing away CMD, ENTRYPOINT, WORKDIR and ENV. Rebuilding Docker's merge rules by hand, and the bug that only showed up on stock images.
A RUN step in a Dockerfile is arbitrary customer code. On Pilots it executes inside a Firecracker microVM like every other workload, driven by buildctl from the host, with no build tier and no daemon sharing the host kernel.
Scale-to-zero usually means a visitor sees a loading screen while something starts. On Pilots the connection is held open while the microVM restores, so the first request is just a slow request.
Deciding where a new machine goes and deciding who rescues a dead host's machines look like the same problem. They are not, and Pilots answers them with different algorithms on purpose.
Firecracker memory snapshots carry raw CPUID, so an image photographed on Intel cannot be restored on AMD. Pilots runs a mixed fleet anyway, with a three-tier rescue that degrades to a cold boot instead of failing.
In a last-write-wins CRDT, two hosts writing one row is not an error. It is a merge, and the loser disappears. The single-writer rule on Pilots, the five exceptions, and why a write-once row is the only shape that is safe for anyone to write.
Every host on Pilots runs the same three processes and serves the whole API from its own local replica of the state. There is no scheduler tier and no managed database. Here is how that works and what you give up for it.
When a host dies on Pilots, every survivor independently computes the same answer about which machines it owns now, from a hash. No coordinator, no consensus round, and the machines keep their URLs.
The machine root on Pilots is durable as of the last flush, at most 60 seconds ago, and that number is a metric rather than a claim. What the periodic flush costs the guest, what it bought self-heal, and the one axis where we are behind.
Most platforms have a machine disk and a network volume, two models with different failure modes. Pilots has one: the root is a read-through cache over object storage, so wiping a host's NVMe costs a download and never a rebuild.
A restored Firecracker microVM gets its memory on demand through userfaultfd, with the pages coming from S3. One round trip per 4KiB page would take seventy minutes for a 256MiB guest, so it does not do that.
NBD_DO_IT parks a thread for the life of the device, and any signal delivered to that thread tears the device down. Go's own scheduler preemption was doing it. The symptom pointed nowhere near the cause.
Chunking a copy-on-write disk by reading the file gives you a rootfs that mounts empty with nothing having errored. Why the dirty bitmap has to come from the block handler, and why allocated extents are not a substitute.
Seven steps take a Firecracker microVM to a durable checkpoint, and every one of them is where it is because moving it made the resume gap worse. Prefaulting before a Full, diffs from mincore, and the 36x speedup that does not survive.
Booting a Linux guest costs the same twenty seconds every time. Pilots restores a memory snapshot of an already-running guest instead, so a new Firecracker microVM is answering in under half a second on real hardware.
Pilots runs Firecracker microVMs as both instant sandboxes for AI agents and durable production services, on one primitive, with no central control plane. The origin story and the three decisions everything else follows from.