Self-Heal With No Leader, No Election, No Human

A host dies. Not gracefully. Somebody trips over a cable, or the kernel panics, or the machine simply stops gossiping and never comes back.

The machines it was running need to come back somewhere else, with the same URLs, without anybody being paged. The usual way to arrange that is a coordinator: something watches the fleet, notices the death, and assigns the orphans to survivors. Then you need that coordinator to be highly available, so you elect it, so you need consensus, so you have a Raft cluster in the middle of your data plane and a new class of incident where the thing that fixes failures is the thing that failed.

Pilots has no coordinator, because the answer to "who takes this machine" is computed rather than decided.

The whole mechanism

Every hostd heartbeats a last_seen timestamp on its own row. A host silent for longer than the dead-host window (30 seconds by default) is considered dead by everyone who can see the row, which is everyone, because state is gossiped to every host.

Each survivor then rescues exactly the slice of that host's machines where

hash(machine_id) mod live_hosts == my_index

That is it. Every survivor runs the same function over the same replicated rows and gets a partition of the dead host's machines with no overlap and no gaps. Nobody tells anybody what to do. Nobody has to agree in the "distributed consensus" sense, because there is nothing to agree about: the answer is a pure function of state they all already have.

A rescuing host recreates the machine from its latest builds in object storage, writes the new host_id, and the URL does not change. The storage model is what makes that possible. The survivor needs nothing from the dead host, because the dead host was never authoritative for anything.

The vendor filter lives inside the same hash

There is one complication, and it is physics rather than design. A Firecracker memory snapshot carries raw CPUID, and a memory image never restores across the Intel to AMD boundary. CPU templates normalise within a vendor, not across one.

So the survivor set is filtered by vendor inside the same hash. Hosts of the memory image's own vendor pool are ranked first, and the winner restores the machine. Only when no host of that pool is alive is the whole live set ranked, and then the winner cold-boots the machine from its own disk instead of restoring it.

That is the three-tier rescue, and the important property here is that adding it did not add a coordinator. It is the same deterministic function over a filtered candidate set, so it still needs nobody to be in charge.

A joining host may act on presence, never on absence

Here is the failure this design has to avoid, and it is the reason for a component that does not exist on most platforms.

A host that has just joined the fleet, or restarted, has a replica that is still catching up. It queries for a machine's rows and gets nothing back. Two completely different situations produce that same empty result: the machine's host is dead and its rows are old, or the machine's host is perfectly alive and this replica has simply not applied its writes yet.

A host that cannot tell those apart, and acts anyway, claims a live host's machines. The claim then merges silently into rows a live host is still writing, which is the single-writer violation in its most damaging form, arriving through the thing that was supposed to fix failures.

So there is a join gate. Until a host's replica has caught up, it may act on its own rows and on the presence of a foreign row, never on the absence of one. It serves its own machines, routes, wakes, meters and answers DNS immediately, and it holds back exactly three callers: self-heal claims, the router's held-request rescue, and autoscaler arbitration.

Caught up means something checkable rather than a timer. No gaps in the bookkeeping table, no SWIM member missing from the version vector, and no live peer ahead of us on any actor. It latches once and never re-closes, because a gate that could re-close would make liveness depend on gossip, and then a gossip hiccup would stop a healthy host from serving.

The fleet battery tests both halves. One section stands up a host whose peers it cannot reach and asserts that it reports itself as still joining, keeps serving its own machines, and claims nothing. The next section runs the same scenario with the gate skipped behind two fault flags and asserts the opposite: the host declares itself caught up, which is precisely the judgement the gated host refuses to make. A negative control matters here because a gate that is accidentally always-open passes every test the positive case has.

Why 30 seconds, and why it is one number

The dead-host window is a single tunable in a single place, and it should be, because it trades two things directly against each other.

Shorter means a faster rescue. Longer means a smaller window in which a gossip stall gets a live host's machines claimed out from under it. Since the root flush made every running machine claimable, the blast radius of that stall is the whole host rather than just its sleeping machines, so the trade got sharper rather than softer.

The recovery is built in. A returning owner finds its rows claimed and kills its own copies of those machines. That is the correct behaviour for a host coming back from a partition, and it is the reason the claim is safe to make at all.

What it looks like from outside

There is a demo on the home page that kills a host and shows the machines landing on the survivors. It is a simulation of the placement rule rather than a live fleet, and it says so, but the rule it animates is the real one: the same hash, the same modulus, the same absence of a leader.

What a user sees is nothing. The URL keeps working. A request that arrives during the rescue is held while the machine comes back rather than bounced, which is the same held-request path a wake uses, because a rescue and a wake are the same restore with a different trigger.

Applications with open connections to a rescued machine do have to reconnect. That is what every failover does, and I would rather write it down than let "no human action" imply "no observable event". The platform recovers on its own. Your connection pool still has to notice.

Every note, or install the CLI and try the thing this one is about.