Skip to content
pilots
Dashboard

← All notes


Vivek

  • ai-agents
  • checkpoints
  • sandboxes
  • mcp

Give the Agent an Undo Button

An AI agent about to run a migration, an upgrade, or a command it just invented is one step from a broken machine. A checkpoint on Pilots restores in place, keeping the same URL and the same agent token, so the undo costs nothing to use.


Watch an agent work for an afternoon and you will see the same moment several times. It is about to do something that might not be reversible. A database migration. A package upgrade that touches thirty transitive dependencies. A rm with a path it assembled from two variables. A command it composed from a README it read ten seconds ago.

Humans handle that moment with a commit, a branch, or a snapshot of the VM, and mostly with the knowledge that they can rebuild the machine if they have to. An agent handles it by hoping, unless the platform gives it something better.

The something better is a checkpoint that is cheap enough to take before every risky step and fast enough to restore that using it is not a decision.

In place, on the same machine

The restore is the property that matters, more than the speed.

Restoring a checkpoint on Pilots does not spawn a fresh machine from an image and hand you a new address. It is the same machine row, the same URL, and the same agent token the agent is already holding.

If it were anything else, every client of that machine would have to re-discover where its work went. The agent's session, the editor connected over the workspace protocol, the browser tab on the preview URL, the .internal name a sibling service resolves. A restore that mints a new identity turns an undo into a small migration, and an undo that costs a migration does not get used.

Checkpoints carry a comment to find them by later, and they can be taken from a restored state. So an agent can walk back to a known-good point and try a different branch, which is the shape of what agents actually do: not one linear session, but a search.

The numbers, and where they come from

The budget on dedicated hardware is a resume gap under 500 ms, and the gap is reported on the checkpoint response as resume_gap_ms.

Two things about that number are worth saying, because both affect how you read it.

It is the freeze, not the call. Most of the work happens with the machine still serving, so a client timing the round trip overstates what its users experienced. Reporting the freeze rather than the duration is the honest number and also the less flattering one.

And it is independent of machine size, which is the property that makes this usable at all. A checkpoint copies the disk ranges that were actually written, using the block handler's dirty bitmap, so the pause is proportional to what changed rather than to how big the machine is. An earlier version copied the whole copy-on-write file inside the pause, and the p50 across hosts ran from 409 ms to 2172 ms with outliers past four seconds. An undo that gets slower as the session gets more interesting is an undo the agent stops taking.

The ordering that gets it there is its own post, and it is genuinely the most order-dependent code in the engine.

Durable in the background, and told apart from local

The machine resumes as soon as the changed blocks are copied. Chunking and uploading happen behind it.

Two markers distinguish the states rather than one: the builds existing on this host, which is everything a local rollback needs, and the builds being in object storage, which is what a restore on another host needs. Checkpoint status reports both.

Collapsing them would make every rollback wait for an upload nobody is going to read, which for an agent taking a checkpoint before each of twenty steps is twenty waits for a durability guarantee that the very next operation does not need.

What the agent actually calls

The tools an agent gets are deliberately boring. Create a machine. Exec a command, buffered or streamed. Checkpoint, with a note on it. Restore, by the id that checkpoint returned. Push a file in, pull one out. Deploy a directory.

Exec streaming is a WebSocket with binary frames, and the protocol has one detail worth knowing if you are implementing against it. The exit verdict goes out as a text frame first and a binary one after it, because a client settles on the first verdict it sees and only the text one is untruncated. A signal death is -1, and one byte reports that as 255.

The MCP server that fronts these is hosted by every host in the fleet, so connecting an agent to Pilots is a token and an entry in a config file rather than a process to run locally. There is a --stdio mode that adds the six tools which read your own files, for the operations where the agent needs to push a directory it can see and the fleet cannot.

The thing I would tell an agent author

Design for the search, not the transcript.

The mental model of a session as a straight line from prompt to result is what produces tooling that cannot back up. Agents branch, fail, and retry, and the tooling either makes that cheap or makes it expensive. A checkpoint that costs 400 ms and keeps the machine's identity makes backing up free. A rebuild that costs ninety seconds and changes the URL makes it something the agent avoids, which means it presses on with a broken machine instead, which is the failure you actually see in the transcripts.

Everything else here follows from that. Suspend costs nothing, so the machine can sit between steps. The wake is a held request, so a preview link works whenever somebody clicks it. The guest holds no credential, so forking one is not a security event. The point of all of it is that the agent never has to decide whether an operation is worth its cost.


Every note, or install the CLI and try the thing this one is about.