Lazy Memory: Serving a Guest's Page Faults From Object Storage
A restored Firecracker microVM gets its memory on demand through userfaultfd, with the pages coming from S3. One round trip per 4KiB page would take seventy minutes for a 256MiB guest, so it does not do that.
When a machine wakes on Pilots, its memory is not in the host's RAM. It is a packed file of 4 KiB blocks in object storage, and the guest is running anyway.
The mechanism is userfaultfd, a Linux facility that lets a userspace process handle page faults for a region of another process's address space. Firecracker supports it directly: you load a snapshot with the memory backend set to a Unix socket, Firecracker sends you the fault file descriptor over that socket with SCM_RIGHTS plus a JSON description of the memory regions, and from then on every fault the guest takes is a message on your file descriptor and your problem to answer.
Answering means UFFDIO_COPY, an ioctl that writes the page content into the guest's address space and wakes the faulting thread. Get it right and the guest never knows. Get it wrong and a guest thread hangs for ever, with no error anywhere, because "nobody answered the fault" is not an event the kernel reports to anyone.
Seventy minutes, or not
The naive handler does one range read from object storage per fault. A round trip to the bucket is about 50 ms. A 4 KiB page per 50 ms is 80 KiB per second, and a 256 MiB guest needs about 65,000 pages. That is roughly seventy minutes to fault in a machine, assuming it touches all of its memory.
So the handler does two things instead.
The first is a coalesced prefault: one bulk range read of the packed data file, in a background goroutine, installing pages as they arrive. Four fault workers handle whatever the guest asks for that has not arrived yet. The bulk read is a single large sequential transfer, which object storage is good at, rather than sixty-five thousand small random ones, which it is not.
The second is a recorded fault order. The handler writes the order in which the guest actually faulted to a prefetch.txt beside the memory image, and the next restore of that machine replays that order. A machine restored twice tends to touch memory in a similar order both times, so the replay is a prefetch that is right rather than a prefetch that is merely early.
The replay set is the recorded order first, then the ranges the last cycle's diff stores itself, capped at 64 MiB. The first part is a sequence that matches real access order. The second is a set with no ordering, which is still worth pulling because the machine definitely wrote those pages and will probably read them.
There is one file-handling detail in there that bit me and is now a comment in the source: read the replay file fully before creating the new record file. They are commonly the same path, and os.Create truncates.
The ioctl number is not a number you can guess
UFFDIO_COPY is 0xC028AA03. The 0x28 in the middle is the size of struct uffdio_copy, which is 40 bytes, and that size is baked into the ioctl number itself. Get the struct definition wrong by a field and you are not calling the ioctl you think you are.
And the _UFFDIO_* numbers are not sequential from zero. _UFFDIO_API is 0x3F. If you assume an enum starting at zero, the API handshake fails in a way that does not say "you used the wrong constant".
These are the kind of details that make the handler a port rather than a rewrite, which is what our AGENTS.md calls it, in those words, for exactly this reason.
The four rules that keep a guest thread from hanging
EEXIST is success. Two workers can race to serve the same page. The second one gets EEXIST, which means the page is already there, which is the outcome you wanted.
A short copy is an error, except on hugepages, where it is a resume point. UFFDIO_COPY can copy fewer bytes than you asked. On a normal page that is a bug worth failing on. On a hugetlb page the copy can be preempted mid-page and is never redelivered, so treating it as a failure hangs the faulting thread for ever. The handler resumes from the byte count the kernel reported.
The EAGAIN retry is bounded. An unbounded retry turns a persistent EAGAIN into a worker spinning on a core with a guest thread blocked behind it. That is a host-level problem produced by a loop somebody wrote to be forgiving.
Minor and write-protect faults are counted and still answered. They are rarer than missing-page faults and easy to leave as a default: case that logs and moves on. Leaving one unresolved wedges the guest thread that raised it, and the guest looks alive because every other thread is fine.
One page size for the whole fleet
Guest memory is backed by 2 MiB hugepages when the host is configured for it. The handler accepts a uniform page size of either 4 KiB or 2 MiB, and refuses a mixed region map at the handshake rather than per fault, because a failure at the handshake is a failed restore with a clear message and a failure per fault is a guest that hangs at an unpredictable moment.
The page size is recorded inside every snapshot and cannot be reinterpreted at restore. A host configured differently from the rest of the fleet cannot restore the fleet's machines at all. Not slowly. At all. That sounds harsh and it is the good outcome, because the alternative is a host that restores them subtly wrong.
Hugepages are also why the snapshot path takes diffs from `mincore` rather than Firecracker's dirty-page tracking: dirty tracking forces KVM back to 4 KiB page tables, which would cost the hugepage win everywhere to gain a better diff in one place.
Capacity accounting has to follow the same decision. A host with a reserved hugepage pool counts free memory from HugePages_Free, not MemAvailable, because the reserved pool is excluded from the latter. A host counting the wrong one reports itself full and refuses every rescue sent its way, which is a failure that only shows up when another host has already died.
And swap must be off. A swapped-out page is not resident, and mincore would leave it out of the diff, which is the empty-filesystem class of bug wearing a different hat.
What it buys
The handler also exposes a control socket with a prefault command that installs every page, which the snapshot path calls before taking a Full, for reasons that belong to that post.
Put together, the effect is that a machine's memory is a thing in object storage that a host borrows. The machine wakes in under 200 ms on real hardware, on a host that may never have run it before, with no copy of it anywhere on that host's disk beforehand. That property is what makes "your suspended machine wakes on any host in the fleet" a design rather than a slogan, and the userfaultfd handler is where it is actually implemented.
Every note, or install the CLI and try the thing this one is about.