Publishing an RPO Instead of Saying 'Durable'
The machine root on Pilots is durable as of the last flush, at most 60 seconds ago, and that number is a metric rather than a claim. What the periodic flush costs the guest, what it bought self-heal, and the one axis where we are behind.
"Durable" is a word that means nothing on its own, and every storage system uses it.
The question a person actually has is how much they can lose and when. For a volume on Pilots the answer is nothing, because a write is durable when it returns. For a machine's root disk the answer is up to 60 seconds, and I would rather write the number than write "durable" and let a reader assume something better.
That number is a Recovery Point Objective, which is the standard term for how far back a failure can throw you. It is published, it is tunable by the operator through PILOT_ROOT_FLUSH_INTERVAL, and the end-to-end battery measures it instead of taking it on trust.
What the flush actually does
Every 60 seconds by default, a running machine takes a disk-only checkpoint. Pause the guest, read the block handler's dirty bitmap, copy the ranges it names, resume, then chunk and upload in the background.
It is the checkpoint sequence with the memory half removed, and its cost is bounded by construction: only the blocks written since the previous flush are copied, so a busy machine's flush is proportional to how busy it has been in the last minute, not to how big its disk is.
Two metrics come out of it. pilots_root_flush_lag_seconds is how far the bucket trails the disk. pilots_root_flush_pause_seconds is what the guest paid, with an SLO of p99 under 25 ms across a fleet.
The battery asserts the realised lag against the 60 second window, that the bulk of pauses are under 25 ms, and that none of them approaches a whole-disk copy. Note what it does not assert: the p99 itself. A dozen flushes over short-lived machines cannot measure a p99, because the first flush of each machine copies the whole overlay and is an outsized share of that sample. An assertion that cannot be true of the sample it runs on is an assertion that will be turned off, and a turned-off assertion is worse than none.
The flush changed what a rescue is allowed to do
This is the part that surprised me, because I added the flush for durability and the bigger effect was on self-heal.
Before it, a machine whose row named no durable disk build had to be left alone when its host went silent. There was nothing in object storage to bring back, and the owner might merely be partitioned, still running the guest quite happily. So a claim on such a machine could only mean destruction, and destroying a machine because its host stopped gossiping is not a trade anybody wants.
With the flush on, every running machine names a durable disk within the window. So self-heal claims every machine of a host that has been silent past the deadline, sleeping or running, and cold-boots it from that disk.
The blast radius of a long gossip stall is therefore the whole host rather than just its sleeping machines, and that is deliberate. A returning owner finds its rows claimed and kills its own copies. A shorter dead-host timeout is a faster rescue and a wider window for that stall to hurt, which is why it is one number in one place instead of a policy spread through the code.
Measured against the product it should be measured against
Sprites is the only comparable product, and I want to compare on both axes, including the one where we lose.
They ship write-back for their root too, and they do not publish a window. Their docs say the filesystem "syncs to it continuously, not as a snapshot taken at hibernation", and that "durability is a property of the disk rather than an event you trigger". Also, in the same body of writing, that "all storage on a Sprite shares this 'eventual durability' property". The only interval published anywhere is "every second or so", and that is for the SQLite virtual filesystem's write buffer rather than for the disk.
So on the knowable window, we are ahead. A number you can read and plan around beats continuous-but-unspecified, and this whole post exists because I think a published RPO is a feature.
On the other axis we are behind, and it is written into the architecture doc so that a reader does not have to go looking.
Their flush never stops the guest. Their own description is that "the storage flush happens as a background write-back rather than as a stop-the-world copy of RAM, so putting a Sprite away is not a thing you wait for", and a checkpoint "runs copy-on-write, so it's fast and doesn't interrupt the Sprite".
A Pilots flush takes a real pause. It is bounded, it is measured at p99 under 25 ms, and 25 ms is inside the noise of an HTTP request. But it is not zero, and zero is what a copy-on-write generation split inside the block cache would buy.
Why that split is not done yet
Because it is a rewrite of the block cache and of the network block device handler's control path, and that handler is explicitly a port rather than a rewrite in this repo's rules. The reason for that rule is four separate kernel-ABI landmines that each cost a day, in a component where the failures do not point at their causes.
So the sequence is: ship the pause, measure the pause, and let pilots_root_flush_pause_seconds decide whether the follow-on is ever worth its risk. If the p99 across a real fleet stays inside the noise of a request, the rewrite buys a number nobody can feel. If it does not, there is a metric saying so, with the machines and the workloads that produced it.
That is the general shape I try to hold to. Publish the number, measure the number, and let the measurement decide what gets rebuilt. The alternative is rewriting a kernel-adjacent component on the strength of an adjective.
Every note, or install the CLI and try the thing this one is about.