The whole idea, before any of the detail

The rest of this page is written for somebody who already knows what a page fault is. This section is not. Read it and you can follow every figure below, and each section repeats its own idea in ordinary words underneath the technical version.

pilots runs your code inside a tiny simulated computer. Not a shared container with a fence around it, an actual separate machine with its own kernel, which is what makes it safe to hand one to a stranger or to an AI agent that is about to run something reckless. The trick is that starting one does not mean booting one. The system keeps a photograph of a machine that has already finished starting up, and a new machine is that photograph brought back to life. Nothing boots, so nobody waits the 20s that booting and settling actually takes.

The same photograph is how a machine goes to sleep and comes back. Stop touching it and it is captured and put away, costing nothing. Send it a request and the request waits, for about as long as a slow web page takes to load, while the machine is brought back exactly where it left off. Its address never changes through any of this, which is the part everything else is arranged around.

The unusual decision is that there is no head office. Most platforms have one place that decides where things run and one database that knows what is true, and if that place is having a bad day, nothing works. Here every server runs the identical software, keeps its own complete copy of the shared facts, and can answer any question about any machine in the system. They tell each other about changes constantly, the way a rumour spreads, so nobody has to ask permission to act.

Why anyone would build it this way

A head office is a single thing that can be down. Removing it means no request ever depends on a particular server being alive, and a server that loses contact with the others keeps working rather than freezing. It also means adding capacity is handing the system an IP address, because there is nothing to register with.

What it costs

Without one authority, two servers can briefly believe different things and neither is told it is wrong. There are no guarantees of the kind a normal database hands out for free. Most of the awkward machinery on this page is the price of that, paid in advance and on purpose.

Where the real copy lives

In shared object storage, not on any one server's disk. The local disk is only a cache. The design is tested against a blunt question: wipe any server completely, and has anything been lost. The answer has to be no.

What a fleet is, and what it is missing

Adding capacity is running one script against an IP address. There is nothing for the new host to register with, because the thing it would register with does not exist.

Every host runs the same 3 processes, holds a full replica of the fleet's state, and can answer an API call for a machine it has never run. The dashed edges are gossip rather than requests, so a partitioned host keeps serving out of its local replica. Object storage holds the only authoritative copy of machine state, which is what makes the arrows between hosts optional rather than load-bearing.

The three that are not there

A platform this shape normally has a scheduler tier deciding placement, a managed database holding the truth, and a load balancer in front terminating connections. Each is a thing that can be down while every host is up. Removing all three is not an optimisation, it is the constraint the rest of the design is bent around, and most of the awkwardness on this page is the bill for it.

Coordination without a coordinator

Where something genuinely needs a single actor, ownership is computed rather than elected. Every host runs the same function over the same replicated inputs and reaches the same answer without exchanging a message about it. That covers name allocation, which host builds an image, and which survivor rescues which machines.

deterministic ownership

owner(key) = hash(key) mod live_hostssorted identically on every host, or it does not tile

One host, and the whole data plane is on it

A host runs 3 processes. One of them is the entire product: the API, the router, the microVM supervisor, the storage client, and the self-heal loop are packages in a single Go binary, and that is why waking a sleeping machine is a function call.

Nothing above talks to another host to serve a request. The router resolves a name out of the local replica, and if the machine is here it proxies straight into its namespace. The guest agent listens on port 3001 inside every microVM and is how exec, terminals, and arbitrary application ports are reached. The rootfs is a private copy bind mounted onto one constant path, which is what lets a snapshot taken here restore anywhere.

Every host serves this

There is no separate admin API and no host that answers more than another. A request for a machine on the other side of the fleet is served by whichever host received it, which is also what makes the wildcard DNS record legitimate rather than a trick.

the public surface, on every host

POST   /v1/machinesPOST   /v1/machines/:id/execGET    /v1/machines/:id/exec/streamPOST   /v1/machines/:id/checkpointsPOST   /v1/checkpoints/:id/restorePOST   /v1/machines/:id/suspend | wakePOST   /v1/buildsPOST   /v1/services/:id/deploy | rollbackPOST   /v1/machines/:id/promoteGET    /v1/hostsbearer auth, checked against the local replica

State that merges instead of failing

Fleet state is a set of CRDT tables gossiped between hosts, so every lookup on the request path is a read against a local disk. What that buys is availability. What it costs is every guarantee a database normally provides, and the table below is how that cost is paid.

Each replicated table, the only host permitted to write it, and what it holds
Table	Writer	What it holds
machines	the host running it	Everything about one microVM: its name, its owner, its lifecycle knobs, the builds it was last captured into, and the moment it was last touched.
hosts	the host itself	Free capacity and a heartbeat. The heartbeat is what the self-heal loop reads, and the capacity is advisory, because a host is the final authority on whether it accepts a placement.
checkpoints	the host running the machine	Named, sequenced, and pointing at the two builds that reconstitute the machine at that moment.
services	the host running it	The production face: replicas, the current release, a health specification, plain environment values, and sealed ones.
releases	the host running the service	One rootfs build plus whether it was ever observed healthy, which is what a rollback selects against.
volumes	the host it is mounted on	Where a volume lives in object storage, and which machine currently has it.
api_keys	any host, on an admin-scoped request	Key hashes only, replicated everywhere so that every host authenticates against its own disk. Each row is written once, which is what makes any host a safe writer.
api_key_revocations	any host, on an admin-scoped request	A tombstone per revoked key. It only ever appears and never changes, so no two writers can disagree about it.
tenancy	the host writing the object row	Which org owns each machine, service, and volume. Written once, before the object row it names, so a create that dies partway leaves an owner and never an orphan.
org_quotas	any host, on an admin-scoped request	Ceilings per org on machines, cores, memory, volume space, and concurrent builds. One logical writer per row, so the merge has nothing to corrupt.
repo_links	any host, on an admin-scoped request	Which repositories an org may have the fleet fetch through its GitHub App. Keyed by the org and the repository together, written once, and read from the local replica before any build names a repository by name.

Nothing enforces that column. Two hosts writing one row does not conflict and does not error, it merges, and the loser disappears with no trace anywhere. That is why the writer is a property of the design rather than a constraint in a schema, and why the exceptions are enumerated rather than left to judgement.

Merges happen per column, so a returning owner can heartbeat a row another host has already claimed. Two rules make that safe. The rescuer writes owner and state in a single transaction, and on every tick a host kills any machine it is locally running whose row now names somebody else. Nothing votes, and nothing waits for a quorum, so a host that is wrong is corrected by reading rather than by being told.

Rows are never deleted

A delete racing an update can resurrect the row through the merge, so a destroyed machine is marked destroyed and collected later by a reaper. The same reasoning applies to revoking an API key.

Names are not unique

There is no uniqueness constraint to lean on, so during a membership change two hosts can briefly both believe they own a name. The router resolves a duplicate by taking the lowest machine id and logging it loudly.

Schema changes are a fleet operation

Altering a live table makes the CRDT layer backfill every row, which is a gossip storm rather than a migration. Evolution means a new table and a period of reading both.

Gossip travels as QUIC inside the encrypted mesh, with the datagram size pinned to 1232 bytes: the smallest the underlay can be, rather than the largest this host happens to support. Left to discover its own path size, it overestimates across a mixed underlay and drops gossip in a way that looks exactly like a cluster flapping at random.

One request, three endings

The router is a package in the binary that supervises the microVMs, so the branch where the machine is asleep is not a distributed operation. The connection is held, the machine is restored underneath it, and the response arrives late rather than never.

The lookup is a read against a replica on the same disk, so choosing between the three branches is not a network operation and cannot fail because another machine is down. Two names can briefly collide during a membership change, since the state layer has no uniqueness constraints, so the router picks the lowest machine id and logs it rather than erroring or choosing arbitrarily. Suspension needs both of its conditions: the 60s timer has to expire and nothing may be in flight, because an agent halfway through a build generates no HTTP traffic at all.

The bucket is the machine, and the disk is a cache of it

Machine state lives in object storage as content-addressed blocks, and the host's disk is a cache of it. That is true of a running machine's root disk and of a volume, not only of a snapshot at rest. The design test is blunt. Wipe any host's disk and nothing is lost.

The dashed arrow and the solid one are the two durability tiers. A volume write is uploaded as it is made. A root write reaches the bucket at the next flush, at most 60s later, or sooner at a checkpoint or a suspend. The middle column is rebuilt from the right-hand one on demand, which is the design test stated as a picture: a host that loses its disk loses speed for a while and nothing else.

No machine owns a copy of its disk

A create does not copy the template. It restores over a build that every machine on the host shares, and only the machine's own writes land in a file of its own, as dirty blocks. A read that the machine never wrote falls through to the template, and a host that has never held that template serves it from the bucket range by range while a background pull fills the cache. The machine attaches at once and its reads do not wait for the copy. A marker file, never a file size, is what says a cached build is whole, because an interrupted pull leaves a full-length file of holes.

Two promises, stated rather than implied

A volume is durable per write: the write is uploaded as it is made, and write-back buffering is deliberately off. A machine root is durable as of its last checkpoint, suspend, or root flush, whichever is most recent, and the flush runs at most 60s apart. A flush pauses the guest, reads which blocks changed, copies only those, resumes, and uploads behind the resume. The pause is bounded by what was written since the last flush and budgeted at 25ms, which is small and is not zero. Data that cannot afford to lose that window belongs on a volume.

Why the cache earns its place

The bucket is a network away even inside one datacentre. One upload stream from a host measured 55 MB/s, which is fine for the truth and useless as a root disk. So every hot read is served from NVMe, the bucket is read only on a miss, and uploads happen behind the guest rather than in front of it. The cache makes the design fast. It never makes it correct, which is why it can be thrown away.

What a dead host costs now

Because every running machine names a durable disk within that window, self-heal claims every machine of a silent host, the running ones as well as the sleeping ones. A sleeping machine wakes elsewhere from its snapshot. A running one cold-boots elsewhere from its last flushed disk, keeping its id, its name and its address. The price is stated too: a gossip stall long enough to look like a death now affects a whole host's machines, and a returning owner finds its rows claimed and stops its own copies.

Memory and disk are both cut into 4KiB blocks. The header carries 64 bytes of metadata and then 40 bytes per mapping, so a machine that changed nothing produces a header full of pointers and a data object of zero length, and its checkpoint skips the upload entirely. Chains stop at 2 levels because an unchanged range names a logical offset rather than bytes, and a grandparent reference is caught when the header is parsed instead of surfacing later as a page that resolves to the wrong content.

A machine is pinned to its template

A diff's unchanged ranges name a logical offset rather than bytes, so they mean nothing except against the exact build they were encoded against. The machine row therefore records which template it was created from, and every later capture and restore uses that one, never whichever template the acting host happens to hold. The two differ routinely. A host restoring a machine whose template it lacks downloads it, which is possible precisely because builds are content-addressed.

The filesystem underneath stopped mattering

An earlier design copied the template at create and copied the whole write layer inside the checkpoint pause, and both were cheap only on a filesystem that shares extents. On one that does not, nothing errored. The copies silently became real ones, and the checkpoint pause grew with the size of the machine, which is the one property a checkpoint cannot have. Serving the root as a block device removed both copies. A checkpoint now copies the changed ranges alone, so the pause follows the writes rather than the disk on every filesystem, and extent sharing is a convenience for the one template copy a host makes.

The first two steps are outside the pause because measuring put them there. Draining the previous capture stops it competing with Firecracker's snapshot write, and making memory resident first turns a first checkpoint of 5.8s into one of 450ms, since Firecracker reads all of guest memory to write a snapshot and any page still lazily backed would fault through the handler with the guest frozen. A suspend differs from a checkpoint here: it chunkifies and uploads before killing the VM, because the kill removes the jail the snapshot lives in.

Durability is two separate signals, and collapsing them would be expensive. One says the builds exist on this host, which is everything a local rollback needs. The other says they are uploaded, which is what a restore anywhere else needs. A single flag would make every rollback wait for an upload it is never going to read.

In plain termsThe server's own disk works like a desk, and the bucket like the filing cabinet behind it. Whatever is being worked on sits on the desk because reaching it is quick. The cabinet holds the real copy of everything, so a desk that is cleared, or lost, costs a walk to the cabinet and nothing more. Files in the cabinet are kept the way a photo album stores a burst of near-identical photographs. The first one is kept in full. Every one after it is kept as the handful of details that differ, plus a note saying the rest is unchanged. A machine that sat idle therefore takes almost no space and almost no time to save, and one that never wrote to its disk skips being saved at all. The last figure is about a different confusion. Saving a machine takes a while, but the machine is only frozen for the middle part of it, so timing the whole operation from outside makes the pause look several times worse than anyone inside it actually experienced.

The guest starts before its memory arrives

A restore does not wait for a memory image or a disk image to land. It starts the guest and answers the faults, which is the difference between a wake measured in milliseconds and one measured in the size of the machine.

The page-fault handler runs 4 workers and answers faults by copying pages into the guest's address space. The accented edge is why it works at all: without one coalesced range read of the packed data, a cold restore costs a separate round trip for every 4KiB block, and on a hugepage host a single fault covers a great many of them. The block handler reads through to the template and only consults its own cache where the machine genuinely wrote, which is what the dirty bitmap records. Inferring that from the file's allocated extents instead is not a substitute, because a filesystem may report data where there is a hole.

Four things the kernel does not forgive

Both handlers are ports rather than rewrites, because what they encode is kernel behaviour that is expensive to rediscover. These four cost the most to learn.

1

A signal delivered to the wrong thread kills a restore

The block handler parks in a kernel call for the life of the device, and any signal delivered to that thread makes the kernel tear the device down. The Go runtime preempts goroutines with a signal, so roughly one restore in four died. The symptom is thoroughly misleading: the attach succeeds, the kernel logs a capacity change, the size returns to zero, and the caller times out against a device that reports no owner and looks free.
2

A disconnected device wedges the host, not the machine

A handler blocked in that same call never reaches its own cleanup, so the device has to be disconnected by the parent before the handler is killed. Get the order wrong and the microVM process blocks uninterruptibly with a dead device until the host reboots.
3

An empty diff is an empty object, and a range read of it fails

A machine that wrote nothing produces a zero-length data object, and a ranged read against it returns a not-satisfiable status. Treating that as an error kills the wake. It means zeros, and the cache is marked accordingly.
4

Retrying forever is worse than failing

A page copy that returns a retryable error is retried a bounded number of times, because an unbounded retry turns a persistent one into a worker spinning on a core with a guest thread blocked behind it.

Every guest has the same address, deliberately

Nothing host-specific may enter a snapshot, and the two places it otherwise would are networking and file paths. Both are solved by making the guest's view constant and keeping everything that genuinely differs outside the image.

A guest resolves a peer by name against a responder on its own tap, answered out of the local replica, scoped to the peer's app and filtered to healthy ones. The address it gets back belongs to the owning host, not to the peer, and the translation happens in the root namespace because every guest sources from the same address and classifying by source is impossible by construction. Two prefixes rather than one widened prefix is what makes the tenant boundary structural: guests may only ever reach the machine range, and hostd only ever listens on the host range. Each host carries 1024 slots. The answers carry a near-zero lifetime, because a rescued machine lands on a new host with a new slot and its address changes.

Two consequences worth stating plainly. A machine rescued onto another host takes a new slot, so its peer address changes, which is why name answers carry a near-zero lifetime and why a connection pool holding an open socket to the old address simply breaks. Recovery covers the platform, not an application's own connections. And because addresses are derived from a host's key, rotating that key readdresses every machine on it, so the host is drained first or every one of them takes a reset.

A Dockerfile becomes the same thing a sandbox starts from

The production face is not a second system. A build produces an ordinary content-addressed template, a release points at one, and starting a replica is the restore path this page has already described.

Builds run on whichever host a hash of the payload picks, in a rootless builder under its own unprivileged user and its own resource slice, with the layer cache shared through object storage so a rebuild is warm on any host. The output is not a special deployment artifact. It is a generation-zero template exactly like the one a sandbox is created from, which is why a replica starts by restoring rather than booting, and why the sign-off budget for a replica or a rollback is <1s rather than a boot time. A service with no domain still health-gates and still rolls back, but it has no concurrency signal, so scaling it to zero is refused at validation instead of being silently redefined.

Three ways in, one pipeline

Direct

The command line tars the local context and posts it. No repository host is involved, which is what the SDKs and the agent flow both use underneath.

Connected to a repository

The webhook endpoint is another route on every host, so any host can receive it and a hash of the repository name picks the builder. There is no continuous integration service in the middle.

A sandbox per pull request

A preview is a sandbox rather than a service, so it idle-suspends to roughly nothing between visits and is destroyed when the request closes.

Build logs are structured rather than a text stream, and that is a product decision. An agent pointed at a repository with no Dockerfile writes one, reads the failing step out of the stream when it is wrong, patches it, and goes again. The loop is what the structure exists for, and it is the flow the final phase gates on.

What drives all of this

The engine finished ahead of the surface over it. What exists on the API today is tenancy, scoped keys, revocation, and quotas. The dashboard and the command line have landed, and the tool server for agents ships inside the command line.

Authentication survives losing any host

A key is minted through the API on any host, under the admin scope, and the first one comes from an operator on a host. The dashboard calls that route and never verifies a key. From then on every host authenticates against its own disk, so killing the dashboard's host changes nothing about which keys the API accepts. Revocation writes a tombstone rather than deleting a row, for the resurrection reason above.

An agent is a first-class principal here. An API key is all one needs, scopes on the key bound what it can do, and the tool server for agents reads the same credentials file the command line does.

the guest agent, inside every machine

GET  /healthPOST /init          set the wall clock after a restorePOST /exec          buffered, with cwd, env and userGET  /exec/stream   binary frames: stdout, stderr, exitGET  /terminal      a pty over a socketX-Pilot-Proxy-Port  reach any port the app listens onrestoring a snapshot leaves the clock frozen, so /init is not optional

The frame protocol is byte-compatible with the sandbox platform this replaces, so a client written against that one drops in. Streaming exec has to support a closed input, because an agent process given an open one waits on it forever.

The nine words this page leans on

A technical page usually loses a reader on vocabulary rather than on ideas. These are the terms doing the work above, defined without pretending the simple version is the whole story.

microVM: A whole computer, simulated in software, with its own kernel and its own memory. Heavier than a container, which shares the host kernel with its neighbours, and much lighter than an ordinary virtual machine. The isolation is real hardware-assisted isolation, which is why untrusted code can be run in one.
snapshot: A byte-for-byte copy of a running machine, its memory included. Restoring one does not start the machine, it continues it: the programs inside were mid-sentence and pick up mid-sentence. This is why starting a sandbox is fast and why a suspended machine costs nothing while asleep.
control plane: The coordinating brain a platform usually has: a scheduler deciding where things run, a central database holding what is true, a load balancer at the front. pilots has none of the three, which is the claim the rest of this page is spent paying for.
gossip: Instead of asking a central database, every machine keeps its own full copy of the shared facts and continuously tells its neighbours about changes. Reading is instant and local. The price is that two machines can briefly hold different answers.
a CRDT, and last-write-wins: The rule for combining two copies of a record that were edited independently. Here the later edit wins, field by field. Nothing errors when two hosts edit the same record, which sounds convenient and is actually the most dangerous property in the system.
content-addressed storage: Data is stored in fixed-size pieces, and a piece that already exists is referred to rather than stored again. A machine that changed almost nothing therefore uploads almost nothing, because most of its pieces are still the ones it started with.
a read-through cache: A nearby copy that is filled by being asked. A read that finds the piece locally is served at once, and a read that does not fetches it from the real store, keeps it, and answers. Nothing in it is the only copy of anything, so it can be emptied at any moment at the price of speed. Here the nearby copy is the server disk and the real store is the bucket.
a page fault: What happens when a program reaches for memory that is not actually loaded. The processor pauses that program, someone supplies the missing piece, and it carries on with no idea anything happened. pilots uses this to start a machine before its memory has finished arriving.
a namespace: A private view of part of the system, given to one process. A machine here gets its own network and its own filesystem view, so it can believe it has an address and a disk path that every other machine also believes it has.

Measured on metal, and held against its budgets

The fleet exists now, so these are its numbers rather than a laptop's. They were printed by the same battery, run on one of the production hosts, and they are reported whether or not they flatter the design. Two of the budgets below are not met yet.

What the battery printed on a production host

468ms create, median on metal

302ms wake, median on metal

403ms checkpoint resume gap, median on metal

352ms wake of a real site through its public address, median, network removed

The budgets sign-off holds it to

<500ms create

<200ms wake

<1.5s promote a sandbox to a service

Create is inside its budget, narrowly. Wake is not: 302ms against <200ms. The last readout in the first row is the same wake as a visitor meets it, a request to a real site that had gone to sleep, with the network's share removed by subtracting a warm request from the cold one. Promote has not been timed on the fleet. The root flush is the other miss. Every pause one host recorded was under 250ms, and only a minority were inside the 25ms budget.

These hosts are older desktop-class processors with the bucket a network away, and the development rig that produced the earlier figures was a newer processor with object storage on the same machine. The rig was faster on every line, which is the opposite of what the page used to predict, and it is why a budget is only worth stating against the hardware it will be sold on.

The largest single change since the engine closed is guest memory backed by 2MiB hugepages. On the rig, the same battery's checkpoint resume gap fell from 3726ms to 300ms, because the page size is recorded in every snapshot and a host that disagrees with the fleet refuses to restore rather than restoring slowly.

The results this design was built to produce are about correctness rather than latency, and those two are still rig figures. On a three-node rig, hard-killing the host that owned a machine returned it on a survivor in 125s with the same address and the disk intact, and a fourth host joined and started taking traffic 15s after one command. Neither has been repeated by killing a production host.

what a run reports

$ PILOTS_E2E=1 npm testcreate, exec, checkpoint, restore, suspend, wake, destroyno orphaned processes, namespaces, slots or portsthe same battery, run against any host in the fleetlater phases add assertions and never retire earlier ones

Everything drawn on this page is written down in full in the design document, and the code implementing it sits beside it. If a figure here disagrees with the repository, the repository is right and this page is a bug. The board tracks what is left (opens in a new tab).

The whole thing, drawn