A Sparse File Cannot Tell You What It Wrote
Chunking a copy-on-write disk by reading the file gives you a rootfs that mounts empty with nothing having errored. Why the dirty bitmap has to come from the block handler, and why allocated extents are not a substitute.
The bug looked like this. Take a machine, write some files in it, checkpoint it, restore the checkpoint. The restore succeeds. Firecracker is happy, the guest resumes, the agent answers its health check. You open a shell and the filesystem is empty. Not corrupt. Not missing some files. Empty, like a machine that never ran, with a clean dmesg and no error anywhere in the host logs.
It took an embarrassingly long time to find, and the cause is a property of sparse files that I knew and had not connected to the thing I was doing.
A hole and a zero read back the same
Each machine's disk is a copy-on-write overlay. The template underneath is shared by every machine on the host, and the machine's own writes land in its own sparse file on top. Sparse means the file has holes: regions that have never been written occupy no space, and reading them returns zeros.
Now chunk that file to make a diff. The obvious implementation walks the file, compares each 4 KiB block against the corresponding block of the template, and records the ones that differ.
A block the guest never wrote reads back as zeros. A block the guest deliberately zeroed also reads back as zeros. There is no bit anywhere in the file that distinguishes them.
So the diff records every untouched block as "this machine wrote zeros here", and a restore faithfully reproduces that: a disk where everything the template had has been overwritten with nothing. The guest mounts it, finds a filesystem, and that filesystem is empty. Every layer did exactly what it was told.
The bitmap lives where the writes happen
The fix is that the thing which knows what was written is the thing that did the writing. The network block device handler serving the machine's disk keeps a roaring bitmap of dirty 4 KiB blocks, because it has to: a read only hits the overlay cache if every block it covers is dirty, otherwise it falls through to the template.
That bitmap already exists for correctness of reads. Chunkification reads it over the handler's control socket, and the copy then covers the ranges the bitmap names rather than the ranges the filesystem happens to have allocated.
This is also why the snapshot sequence reads that bitmap while the guest is still paused. A bitmap read while writes are in flight describes a disk state that never existed at any instant, which is a subtler version of the same class of bug: the pieces are each real and the whole is not.
Allocated extents are not a substitute
The tempting shortcut is SEEK_HOLE and SEEK_DATA, or FIEMAP, and asking the filesystem which regions of the file are allocated. It is right there, it needs no control socket, and it is wrong in one direction that matters.
A filesystem may report data where there is a hole. Nothing in the interface promises that allocation is exactly the set of blocks you wrote: filesystems preallocate, they align, they merge, and a delayed allocation can materialise a region your application never touched. The bitmap is authoritative about writes. The extent map is authoritative about storage, and those are different questions that agree most of the time.
Most of the time is the problem. A shortcut that is correct in testing and wrong under fragmentation produces the empty-filesystem bug again, on a customer's machine, months later.
What the chunk format does with all this
A build's header is 64 bytes of metadata followed by 40-byte mappings. Version, block size, size, generation, build id, base build id, and then one mapping per range with its offset, length, source build id and offset into that build's packed data.
Two things in that format are worth calling out, because both are load-bearing.
A mapping with a nil build id means a zero-filled gap. That is how a diff says "this range is zeros" on purpose, which is exactly the statement the naive chunker was making by accident about the entire disk. Having an explicit encoding for it means the honest case is expressible and the accidental case is not.
Diff chains are exactly two levels. A template, and one per-machine diff pointing at it. A grandparent reference is a hard error rather than a supported case. Chains of diffs are how a storage layer acquires a read amplification problem it cannot back out of, and the restore path has one job, which is to be fast.
The pairing check that turns a silent stitch into a failure
There is a second version of this class of bug, and it is worth the paragraph because the fix is one function.
A diff's unchanged ranges name a logical offset rather than bytes. They only mean anything against the exact build they were encoded against. So a machine records which template build its memory and disk were created from, at create time, and every later capture and restore uses those recorded ids rather than whichever template the acting host happens to hold.
Those two differ routinely in a healthy fleet. The golden template gets rebuilt, or a host that has never held one mints its own with fresh ids. A host restoring a machine whose template it lacks materialises that template from object storage, which is a download rather than a rebuild, because builds are content-addressed.
block.SetParent verifies the pairing and refuses a mismatch. Without that check, a machine stitched onto the wrong parent restores successfully and reads garbage in the ranges the diff did not cover. With it, you get a failed restore and a log line naming both ids.
I will take a loud failure over a successful restore of a machine that is quietly wrong, every time. The whole class of bug here has one shape: a layer that cannot distinguish "nothing" from "zero" hands you something that looks fine.
Every note, or install the CLI and try the thing this one is about.