A Platform With No Control Plane, and What It Costs
Every host on Pilots runs the same three processes and serves the whole API from its own local replica of the state. There is no scheduler tier and no managed database. Here is how that works and what you give up for it.
Draw the architecture of almost any container or VM platform and you get two tiers. There is a control plane, which holds the state, takes the API calls and decides what runs where, and there is a fleet of workers that do what the control plane says.
That split is sensible. It is also the thing that is down when your platform is down. The control plane is a distributed system with a database, a leader, an upgrade path and a capacity story, and it exists to manage another distributed system. Most of the operational pain I have watched teams take on comes from the middle tier rather than from the machines actually running the work.
Pilots does not have one. Every host runs the identical stack, and every host serves the entire API.
Three processes, and that is the list
On a Pilots host there are exactly three things running:
hostd, which is ours, in Go, and is the entire data plane. The API, the router, TLS, DNS, the idle monitor, the autoscaler, the builder driver, metering, self-heal. All of it.corrosion, Fly's gossip-replicated SQLite, run as a binary.firecracker, spawned once per machine.
Systemd manages the first two. There is no message queue, no etcd, no Postgres, no agent-plus-controller pair, and no fourth thing that turns out to be load-bearing.
State lives in Corrosion, which replicates SQLite tables between hosts over SWIM gossip using cr-sqlite's CRDT extension. Every host holds a full local replica. A lookup during a request is a read from a local SQLite file, in microseconds, with no network call in the request path at all.
That last property is the one I care most about. The router resolving a hostname to a machine, the wake path deciding whether to hold a connection, the DNS responder answering a .internal name: none of them can be slow because another host is slow, and none of them can fail because another host is down. They read local disk.
Any host answers, so no host matters
A create, a deploy, an exec stream, a metrics scrape and a certificate challenge are all answered by whichever host the request happened to reach. The wildcard DNS record points at every host's IP. There is no designated API server, so there is no API server to lose.
Adding a host is one command with an IP. It joins the gossip mesh, catches up, and starts taking traffic. Nothing registers it anywhere, because there is nowhere to register.
The shape is borrowed, and I want to be clear about the lineage: uncloud is Apache-2.0, it is the source of this arrangement, and reading it is a better use of an hour than reading most architecture posts including this one. What Pilots adds is the microVM engine underneath and the parts a multi-tenant product needs, but the fleet shape is theirs.
The cost, stated plainly
A CRDT is not free. Last-write-wins merge means no uniqueness constraints and no cross-host transactions. The correctness burden does not disappear; it moves out of the database and into the code that writes rows.
The rule that holds it together is that a host writes only rows describing its own machines. Breaking it does not produce an error. It produces a silent merge that corrupts state later, somewhere else, in a way that is genuinely hard to trace. That invariant has its own post, including the narrow set of exceptions and why each one is safe.
Anything that genuinely needs uniqueness or a single actor uses deterministic ownership instead: hash(key) mod live_hosts, which every host computes identically from state they all have. Name allocation works that way. Self-heal slices work that way. There is no election anywhere in the system, because there is no role to elect anyone to.
And a replica that is behind cannot be trusted to reason about absence, which is why a joining host holds back its claims until it has caught up. That gate is the other half of making leaderless work, and it is a real component with a real test rather than a comment.
"Coordinators propose, hosts dispose"
There is one more rule that makes leaderless placement work, and it is the sentence I keep coming back to when a new feature needs to decide something fleet-wide.
Any host can propose. Only the target host can accept.
A create ranks the fleet and picks a candidate, on whichever host took the request, from that host's own replica. Two hosts ranking two creates differently is completely fine, because the ranking only proposes. The target host admits the machine against its own free memory, or refuses with a 507, and the ranker tries the next candidate. After three refusals it serves the machine locally and lets its own admission control answer.
Nothing has to be reserved anywhere. Nothing has to be locked. The host that will actually run the machine is the only authority on whether it can, which is both the correct answer and the one that needs no coordination. Placement and rescue are different questions with different answers, and this is why.
Where the seams actually are
I do not want to claim this is free. Three things are harder in this shape than they would be with a control plane.
Schema changes. You cannot ALTER TABLE a live Corrosion table, and you cannot add a column to one that has rows. cr-sqlite backfills every existing row on a column add and gossips the backfill, which took Fly's fleet down twice for about eleven and a half hours in one incident. So a shape change here is a new table plus a dual read, and a table that has ever held a row is closed to column adds. That is written into the schema file as a comment, and it is why several things in this system are side tables rather than columns.
Debugging a merge. When state is wrong, there is no single log to read. The evidence is spread across the hosts that wrote the rows, at the times they wrote them.
Operator tooling has to assume a stale replica. A host whose replica is corrupt or hopelessly behind gets re-seeded from a survivor, and that script stops hostd first, because the reaper kills Firecracker processes with no matching machine row and running it against an empty replica would destroy the very machines the re-seed is trying to save. That ordering is not obvious until you have thought about it for the second time.
None of those are free. I still take this trade, because what I get back is that there is no tier whose failure takes the platform down. The failure mode I was most afraid of is the one that does not exist.
Every note, or install the CLI and try the thing this one is about.