Skip to content
pilots
Dashboard

← All notes


Vivek

  • security
  • credentials
  • tokens
  • snapshots

The Machine Has No API Key, and That Is the Point

A credential baked into a guest is in every snapshot and every fork of that guest. On Pilots a machine asks for a token over a socket that only it can reach, and the request path is the identity.

Give a sandbox an API key and think about where that key ends up.

It is in the guest's filesystem, so it is in every snapshot of that guest. It is in every checkpoint. It is in every machine forked from it. It is in whatever the guest writes to object storage, if the guest ever tars up its own disk. And the moment the platform wants to rotate it, there is a fleet-wide re-issue path that has to be correct on restore, on rescue, on promote and on fork, for ever.

The alternative is that the machine holds nothing and asks.

The socket only that machine can reach

hostd binds a broker on the constant gateway address inside each machine's own network namespace, at 169.254.0.22:3002. Every guest sees the same address, because every guest sees the same network, which is what makes snapshots host-agnostic in the first place.

A namespace is not something a guest can leave. So the only party that can reach a given machine's broker socket is that machine, and the request path itself is the identity.

That sentence is the whole design. No credential is presented, because there is nothing a guest could present that a copy of that guest could not. Any secret you bake in to prove identity is copied by the next fork, and then two machines can prove they are the same one. The namespace boundary is not copyable, so it is the only honest identity a guest has.

Three endpoints:

  • GET /identity says who this machine is and where the API is.
  • GET /token?scope=… returns a signed claim, valid 15 minutes, stored nowhere.
  • GET /secrets returns the values granted to this machine, opened with the fleet key.

That last one is worth pausing on. Those are the secrets that never enter the machine's environment file, so they are in no snapshot and on no disk. The guest reads them over a socket when it needs them.

Deny by default, and a grant you cannot escalate through

With no grant, both /token and /secrets answer 403.

A grant is written by an operator through the machine or service endpoint, and a caller may only grant scopes it already holds. admin is never mintable, by anyone, through this path.

The grants live in their own table, written by the host that owns the row they describe, on the same terms as the other object-row side tables. One logical writer, which keeps it inside the single-writer rule without needing an exception.

A token is a claim, not a row

The token format is pbt1.<claims>.<hmac>, signed with a derived label of the fleet's agent-token secret.

Which means every host can verify one without holding anything new and without a lookup. No token table, no cross-host read, no cache to invalidate. A host that has never seen this machine, this org or this token verifies it from local state alone.

Three things stop a token, and all three are local:

  • Its own expiry, which is 15 minutes.
  • The write-once revocation tombstone the key revoke route already writes. Write-once means any host can write it safely, which is the shape that makes a row safe to share.
  • The machine row being destroyed.

Verification with no lookup is what lets this work on a fleet where the request might land anywhere. A token minted by one host is accepted by another, immediately, with no coordination. The fleet battery asserts exactly that, because it is not observable from the public API.

Two limits, stated rather than implied

I would rather write the boundary than let a reader assume a stronger one.

A compromised host mints any token. It holds the fleet key, so it can sign any claim. That is the same trust class as the fleet key itself, and it is the same limit sealed secrets have. Real untrusted-host secrecy needs a key management service, which is a central service, which is a conversation about the one architectural rule this whole system is built on.

A fork taken within a token's life inherits that token for the rest of its life, in the same org. Bounded by the 15 minutes, and bounded by the fact that a fork is already a copy of everything else the source had. It is not nothing, and a shorter token life is the lever if it ever matters.

Both of those are in the architecture doc in the same words. A security design with no stated limits is either lying or has not been thought about, and I prefer readers who can check my reasoning to readers who are reassured.

What it looks like from the guest

An agent running inside a sandbox curls its own gateway, gets a short-lived token scoped to what it was granted, and calls the API with it. If the machine is forked, the fork asks too, and gets its own token for its own identity.

Nothing was baked in. Nothing has to be rotated on restore, because there is nothing on the disk to rotate. A snapshot of the guest contains no credential, so publishing one is not an incident. That last property is what I actually wanted: the ability to hand somebody a checkpoint of a running machine without thinking hard first.

Every note, or install the CLI and try the thing this one is about.