The Order of a Snapshot Is the Design
Seven steps take a Firecracker microVM to a durable checkpoint, and every one of them is where it is because moving it made the resume gap worse. Prefaulting before a Full, diffs from mincore, and the 36x speedup that does not survive.
A checkpoint on Pilots has a number attached to it, resume_gap_ms, which is how long the guest was actually frozen. Not how long the API call took. The freeze is the part a user feels, and the call is longer than the freeze because most of the work happens with the machine still serving.
Getting that number under 500 ms took seven steps in a specific order, and I want to write down why each one sits where it does, because the order looks arbitrary until you have moved one and watched the number move.
1. Wait for the previous snapshot's upload before you pause
The background half of a checkpoint reads the whole memory image, chunks it, and uploads it. If you start a new snapshot while the last one is still doing that, the upload competes with Firecracker's snapshot write, and Firecracker's write happens inside the pause.
So the freeze a user feels belongs to the checkpoint before the one they asked for. That is a horrible property to debug from the outside, because the slow checkpoint is never the one you are looking at.
2. Make memory resident before you pause, but only for a Full
Firecracker reads all of guest memory when it writes a Full snapshot. With lazy memory behind a userfaultfd handler, any page that has not been faulted in yet gets faulted with the guest frozen, one page at a time, through a handler that is now competing with nothing but itself.
On a first checkpoint that is the entire image. Measured: 5.8 seconds of pause, against 450 ms when the pages are already resident. So the snapshot path asks the handler to prefault everything first, while the guest is still running.
And then the word "only". Under a Diff snapshot, the dirty set is the resident set (more on that below), so prefaulting first turns a Diff back into a Full and throws away the entire benefit of taking a Diff. Same operation, opposite effect, decided by which flavour of snapshot is about to happen. It is the single most order-dependent line in the engine.
3. Reclaim inside the guest, in one exec
Four things, over the guest agent, in a single command: fstrim, sync, drop_caches, compact_memory.
Only sync is required. It is what makes the memory image and the disk image agree about recent writes, and a snapshot whose two halves disagree is a machine that resumes with a filesystem from slightly before its own memory. The other three shrink what the snapshot has to carry at all, and each is tolerated individually, because a guest missing one of those knobs still wants the rest.
4. Pause, then choose Diff or Full by one condition
PATCH /vm Paused, then PUT /snapshot/create.
The flavour is a Diff whenever the local mem.bin is exactly mem_size_mib, and a Full otherwise. That condition is not a heuristic. Firecracker merges a diff into that file in place under exactly that condition, and if the condition does not hold it overwrites the file with a partial image instead. It does not fail. It produces an image whose untouched pages read back as zeros, and you lose the machine's memory one restore later, at a moment unrelated to anything you did.
So the first snapshot of every machine lifetime is a Full. A woken machine has no local image because suspend removes it, which means the rule is not "first ever" but "first since the last suspend", and writing the condition rather than the intent is what keeps that correct.
track_dirty_pages stays off. The diff comes from mincore, which reports which pages are resident, because that is the only flavour that composes with hugepage-backed guest memory. Dirty-page tracking forces KVM to 4 KiB page tables, which costs the hugepage lever everywhere to win in one place.
Measured on Firecracker 1.16.1 over a userfaultfd-backed 512 MiB guest: a Diff takes 78 to 116 ms against 2.8 to 3.5 seconds for a Full of the same paused instant, with the merged image byte-identical to the Full. That is the 36x, and it is real.
The 36x does not survive the steps in front of it
Here is the part I would want to read if I were evaluating this design, so it is in the architecture doc and it is here.
mincore reports residency, not dirtiness. Step 2 prefaults every page before a Full, and nothing evicts a page installed through userfaultfd. So from a machine's first checkpoint onward, mincore reports all of memory as resident, and every later Diff writes nearly all of it.
End to end that is 412 ms against 295 ms. A ratio of 1.4, not 36.
The Diff is never worse than a Full, and it is what makes an O(dirty) pause possible in principle. But the pause is genuinely O(dirty) only for a machine whose memory is not already fully resident. Closing that gap needs a dirty set that is not residency, and upstream Firecracker offers that only through track_dirty_pages, which cancels hugepages. It is the one place this design is knowingly leaving a win on the table, and the honest version of the sentence "we take diff snapshots" has that footnote attached to it.
5. Read the disk's dirty bitmap while the guest is still paused
The network block device handler keeps a bitmap of which 4 KiB blocks the machine has written. hostd reads it over the handler's control socket, and it reads it inside the pause, because a bitmap read mid-write describes a disk state that never existed.
That bitmap is also the reason a copy-on-write file cannot be chunked by looking at the file. That has its own post, because the failure is silent and the reasoning is not obvious.
6. Copy the dirty ranges, resume, then upload
For a checkpoint: copy the ranges the bitmap named, which is O(writes) on every filesystem, and resume the guest immediately. Chunkify and upload afterwards, in the background, behind a semaphore that defaults to one, because unbounded chunkification OOMs hosts.
The memory image is chunkified in place, which is only safe because of step 1. Step 1 guarantees Firecracker cannot overwrite it while that is happening.
A reflink of the whole copy-on-write file stood at this step once. It was the size-dependent half of the pause, and worse than that, reflinking half a gigabyte pinned extents whose allocation cost resurfaced as a multi-second pause a checkpoint or two later. A cost that shows up two operations after the thing that caused it is the worst kind of cost.
For a suspend the shape is different, because the machine is going away: chunkify both halves synchronously, upload, kill the VM, and only then upload the recorded fault order. The handler writes that file as it runs, so it is complete only once that process is gone.
7. Two markers, because durable and local are different questions
A .chunked marker means the builds exist on this host, which is everything a local rollback needs. A .durable marker means they are in object storage, which is what a restore anywhere else needs.
Collapsing those two into one flag would make every rollback wait for an upload it is never going to read.
Three smaller things that each cost a day
The vmstate is uploaded before the machine is killed, because the kill removes the jail the file lives in.
A machine that wrote no blocks skips the rootfs build entirely. There is nothing to diff, and producing an empty diff is work plus a row.
And never call global sync(). It holds the kernel block-device lock, and concurrent suspends serialize behind it for minutes. The guest-side sync in step 3 is inside one guest. The host-wide one is a fleet outage with a very innocent-looking call site.
Every note, or install the CLI and try the thing this one is about.