The Invariant That Corrupts Silently If You Break It
In a last-write-wins CRDT, two hosts writing one row is not an error. It is a merge, and the loser disappears. The single-writer rule on Pilots, the five exceptions, and why a write-once row is the only shape that is safe for anyone to write.
Here is the property that makes a gossip-replicated CRDT pleasant to build on, and the same property that will quietly destroy your state.
Two hosts write the same row. Neither write fails. There is no conflict error, no version check, no retry. The merge rule picks one, usually by timestamp, and the other one never happened. Your system stays available, every write returns success, and some of them were lies.
That is not a bug in Corrosion or in cr-sqlite. It is what last-write-wins means, and it is the right trade for a fleet with no coordinator. But it moves the entire burden of write correctness out of the database and into the code, where nothing enforces it for you.
The rule we hold is one sentence: a host writes only rows describing its own machines.
Why it has to be a rule rather than a check
A uniqueness constraint would do this job in a normal database. There is no such thing here, because a constraint needs a single authority to evaluate it and the whole point is that there is not one.
So the enforcement is review. That is unsatisfying, and I have tried to find a better answer twice. The reason it stays a review rule is that the violating code always looks reasonable in isolation: a host updating a row it can see, in response to something real, with correct-looking logic. The bug is not in the statement. It is in which host is running it.
What you can do is design the rows so that the rule is easy to keep, and that is where most of the actual work went.
A row is safe for anyone to write when it is written once
The insight that unlocked the awkward cases is that last-write-wins has nothing to corrupt if there is never a second write.
So the exceptions to single-writer are not "these tables are special". They are rows that are written exactly once, or that have exactly one logical writer. Tenancy claims, API key revocations, repository links. Each is a write-once fact. Two hosts writing the same fact is not a conflict, because the two writes are identical in every field that matters.
The side tables are the same idea applied to a shape problem. A machine's URL access mode and its broker grants could have been columns on the machines row, and they are separate tables instead, each written by the host that owns the object row it describes. Part of that is the schema rule that a table which has ever held a row is closed to column adds, and part of it is that a narrow table with one writer is easier to reason about than a wide row with several reasons to be updated.
The planned handoff, which is the hard one
There is one case where a machine changes owner while both hosts are alive: draining a host for maintenance. It is the narrowest shape I could find that works, and the checks are the interesting part.
The source host, which owns the machine, suspends it, then writes a write-once offer row naming the target. The target's claim is validated against that row rather than against anything the target believes about the world. Four conditions:
- The offer must name this host.
- It must come from the machine's current owner.
- It must be the machine's newest offer.
- The machine must not be running.
The last two are the ones that are easy to omit and expensive to omit, and neither was in the first version.
Without "newest", a target the source gave up on can arrive late and take a machine that has since been offered elsewhere. The offer row is write-once, so both offers exist, and a stale claim looks exactly as valid as a fresh one.
Without "not running", you get two Firecracker processes serving one machine id on two hosts. Both write rows about it. Both are convinced they are right. The merge picks alternating winners and the machine's state becomes a blend of two divergent histories.
Note what the validation does not do: it does not ask the source host anything. It reads rows. A handshake would need both hosts alive at the same instant, which is the coordination this architecture spends its effort avoiding.
Deterministic ownership, where something does need one actor
Some things genuinely need exactly one actor. Allocating a name that must be unique. Deciding which survivor rescues which machine. Firing a cron job once rather than once per replica.
The answer is always the same function: hash(key) mod live_hosts. Every host computes it from replicated state, gets the same answer, and the one whose index matches acts. Nobody is elected, and there is no lease to renew.
Name allocation is the cheaper half of that shape in practice. A local read of both namespaces before the single-writer row is written, plus a deterministic lowest-id tie-break at read time for the cross-host race a local read cannot prevent. Two machines that somehow claim one name resolve to the same winner on every host, which is the property that matters. Eventual agreement with a deterministic rule beats immediate agreement that needs a round trip.
Never act on the absence of a row
The rule has a second half that took a production-shaped scare to internalise.
A host may act on its own rows, and on the presence of a foreign row. It may never act on the absence of one.
A replica that has not caught up returns an empty result for a machine whose host is alive and busy, and an empty result for a machine whose host is genuinely dead. Those are the same query result and opposite situations. Code that treats "I see nothing" as "there is nothing" claims live machines, and the claim merges into rows their real owner is still writing. That is the single-writer violation arriving through the front door of a feature designed to improve reliability.
Hence the join gate in self-heal, which holds back exactly the three callers that reason about absence until the replica is provably caught up.
The cost of getting it wrong is the reason for the strictness
I am stricter about this in review than about anything else in the repo, and the reason is the shape of the failure rather than its likelihood.
A single-writer violation does not page anyone. It does not show up in a test. It produces a row that is subtly wrong, some time after the write, on a host that was not involved, and the evidence is distributed across the hosts that participated. By the time somebody notices, the merge that caused it is hours of gossip in the past.
That is why it is written in capital letters in the architecture doc, why the exceptions are enumerated rather than described, and why every new table gets the question asked of it before it exists: who writes this, and what happens when two hosts do.
Every note, or install the CLI and try the thing this one is about.