Nothing Boots: Why a Create Is a Restore

Booting a Linux guest costs the same twenty seconds every time. Pilots restores a memory snapshot of an already-running guest instead, so a new Firecracker microVM is answering in under half a second on real hardware.

Boot a Linux guest and you pay for the operating system every single time. On our hosts that is about twenty seconds from kernel entry to systemd reporting system-running, and no amount of tuning makes it two hundred milliseconds, because the work is real. Devices get probed. Services start. The scheduler and the page cache warm up.

Every sandbox product runs into this wall and picks one of two answers. Either you wait for the boot, which is what "provisioning your environment" means on a loading screen, or the platform keeps machines running so that one is always available, which someone is paying for the whole time nobody is using them.

Pilots takes the third answer, which is to never boot at all.

The golden template, built once per host

Each host does the twenty-second boot exactly once. It starts a guest from the golden rootfs, waits for systemd to report system-running, gives it a settle period, and then chunkifies the guest's memory into content-addressed blocks in object storage. That is the golden template: a photograph of a Linux machine that has finished starting.

Every machine created after that is a restore of that photograph. Firecracker loads the vmstate, the guest resumes, and the thing that comes back is indistinguishable from a machine that booted, because it is a machine that booted. It just did the booting once, on a Tuesday, for everybody.

Two details in that paragraph are load-bearing and were both learned the hard way.

The first is the settle. Snapshot a fresh boot too early and you capture a half-converged guest, which resumes into a machine that cannot serve anything because the service it needed was still starting when the shutter clicked. Twenty seconds of settling is not a magic number so much as an empirical one, and the check is the guest reporting system-running rather than a stopwatch.

The second is that the template's disk half is the golden rootfs chunked directly, never a snapshot of the booted machine's disk. The memory image describes a specific disk state. If the disk you restore against is not byte for byte the disk the memory image was photographed over, you have a guest whose memory and filesystem have never met, which fails in ways that are extremely annoying to debug because nothing errors. The filesystem just has content the kernel does not believe is there.

The template does not start your application, on purpose

Here is a consequence that took me a while to see clearly.

A create is a resume. That means PID 1 and everything under it are already running at the instant the machine appears. And you cannot inject environment variables into a process that is already running. Docker's environment: semantics assume the process starts with its environment block populated, and in a restore there is simply no such moment.

So the golden template deliberately stops short of starting the application. It settles the base system and nothing else. The guest agent execs your application afterwards, riding the POST /init poke that already fires after every resume to set the guest's wall clock. That clock poke is not optional either: kvm-clock covers the monotonic clock, but CLOCK_REALTIME comes back frozen at the instant of the snapshot, and a restored guest whose idea of "now" is three days ago fails TLS handshakes in a way that looks like a certificate problem.

The asymmetry is easy to get backwards, so it is written down in the architecture doc in capital letters. Delivery-and-exec happens on create from a template. It never happens on wake. A wake resumes a snapshot in which your application is already running, and re-execing there would restart the very process the guest just restored. Which means changing an environment variable takes effect on an explicit restart or a roll, never through a suspend and wake cycle. A suspend is a freeze, and a freeze has to give you back exactly what it took.

What the restore path actually does

The restore is the same code for all three things that use it. Creating a machine from a template, waking a suspended one, and restoring a checkpoint are one path with different inputs.

Prefetch the vmstate and the recorded fault order from object storage if this host does not already have them.
Set up the network namespace and spawn the userfaultfd handler in parallel goroutines. These two do not depend on each other, and running them concurrently saves 150 to 250 ms of wall clock.
Bind the router's port while Firecracker is starting, rather than after.
PUT /snapshot/load with the memory backend pointed at the handler's socket and resume_vm: false, then PATCH /vm Resumed.
Poke the guest agent at /init with the current time, asynchronously, on a 5 ms retry loop with a 15 second deadline.

The budgets on dedicated hardware are create under 500 ms, wake under 200 ms, and a checkpoint resume gap under 500 ms. On a nested-virtualisation laptop node the same battery holds a looser set, because a laptop cannot do this in 500 ms and pretending otherwise turns the assertion into decoration. The metal tier is asserted only when the operator sets PILOTS_E2E_METAL=1, which is a human saying "this is real hardware". Nothing a host reports about itself can stand in for that.

Memory is not copied, it is faulted in

The restore does not read the whole memory image before the guest runs, which is the other half of why it is fast. Firecracker hands the guest's memory to a userfaultfd handler, and pages arrive when the guest touches them.

Cold, with nothing cached on the host, that could be a disaster. One round trip to object storage per 4 KiB page is roughly 50 ms, and 256 MiB of memory at that rate is about seventy minutes. The handler does one coalesced bulk range read of the packed data file in a background goroutine instead, and records the order in which the guest actually faulted so the next restore of the same machine can replay that order as a prefetch. That is the difference between a design that works and a demo that hangs, and it has its own post.

The part I would not do again in a hurry

An earlier version of this engine copied the golden template's disk on every create using a reflink (a copy-on-write clone that filesystems like btrfs and XFS can do without duplicating data). On a filesystem that supports it, that copy is nearly free. On ext4, which does not, cp --reflink=auto --sparse=always falls back to a real copy, and we measured 134 ms warm and 465 ms cold inside a create that had a 1.5 second budget.

It fit. The problem was the other end. Checkpointing copied the whole copy-on-write file inside the pause window, and that made the pause proportional to the size of the machine instead of the size of what changed: 409 ms to 2172 ms p50 across hosts against a 500 ms budget, with single samples past four seconds.

Size-independence is the property that makes checkpoints usable at all. An agent that checkpoints before every risky step cannot have the checkpoint get slower as the machine gets more interesting. So the root moved behind a network block device served from a shared template, per-machine writes land in a sparse overlay, and a checkpoint copies only the ranges that were actually written.

The host still probes whether the filesystem can share extents and reports it on /v1/health, because it is useful to an operator sizing a box. Nothing depends on the answer any more. There is no degraded tier where the budget relaxes on a host with ext4, and there should not be, because a budget that relaxes to fit the host is how a real slowdown hides in a green test run.

Every note, or install the CLI and try the thing this one is about.