Nothing Secret Goes Into a Gossiped Row
When every row replicates to every host, a secret in a row is a secret on every disk in the fleet. How Pilots resolves secret references client-side, seals what it stores, and what that defends against.
Corrosion replicates every row to every host. That is the property the whole fleet design rests on, and it has one sharp consequence for configuration: anything you put in a row is on every disk in the fleet, and in every backup of every one of them.
So nothing secret may be written to one in the clear.
The fleet shape here is borrowed from uncloud, which takes this seriously and is worth looking at for how the problem behaves rather than as a cautionary tale. Its replicated row for a container is Docker's own inspect output, and the environment is stripped out of it before the row is written. The interesting part is what the stripping missed: a pre-deploy hook's environment travelled in a different field and went out with it, and a managed-DNS API token sits in plaintext in another replicated row.
That is the shape of this problem in one project that was already careful about it. A secret does not reach a gossiped row through the field everyone is watching. It reaches it through the second field that also holds values, added later, by someone who had the first one in mind.
The value never enters the repository
The first half of the answer happens before any request is sent.
A secret://name reference in a compose file is resolved client-side. POST /v1/compose/plan returns a step's secret references as names, never values. The CLI then resolves those names against the operator's own store and sends the values as secret_env on the request that needs them.
The operator's store is written by pilot secrets set and pilot secrets import, and listed by pilot secrets ls, which prints names and digests only. All three are local file operations against a 0600 credentials file. No request is made to any host to manage a secret.
So the compose file in the repository says secret://stripe-key, the plan endpoint returns stripe-key, and the value goes from your own machine to the host that needs it, over TLS, when it is needed.
What the host stores, and how
hostd seals the value with a fleet key from /etc/pilots/config before the row is written.
Non-secret values live in one column, sealed ones in another. No plaintext in a gossiped row, and none in object storage.
Two columns rather than one is a small decision with a useful effect: reading the configuration of a service never accidentally touches sealed material, and a code path that wants to show you your environment can show you the half that is showable without a decrypt it might forget to gate.
The limit, stated rather than implied
This defends against gossip spread, and against anything that reads a database file or a backup. Both are real: a replicated row is the widest possible distribution inside the fleet, and it is the thing that goes wrong first.
It does not defend against a compromised host, because any host with the fleet key decrypts any org's secrets.
The obvious improvement is to seal per host, to the owning host's key. It does not work, and the reason is worth keeping because it is the same reason several other things here look the way they do. Self-heal rescues a dead host's machines onto survivors. A survivor bringing up your service needs its environment. If the value were sealed to the dead host's key, the rescuing host would need plaintext it cannot obtain, and the machine would come back unable to run.
Real untrusted-host secrecy needs a key management service. That is a central service that every host depends on for a request path, which is the one rule this architecture does not break. So the honest position is: sealed against the failure modes that actually happen, with the trust boundary written down, and no pretence that a host you do not trust is a host you can safely run on.
Key custody is the operator's job, and it is the only thing like that
The fleet key is an exception to the rule that object storage is the only truth, and it is the only one.
It is operator-held, supplied out of band to the bootstrap script, and lives only in /etc/pilots/config. Wipe every host and the sealed values are unrecoverable with object storage fully intact.
That is a real loss scenario, and it is stated in the architecture doc rather than discovered. The key is in the same trust class as the SSH key that runs the bootstrap: if you lose that, you do not have a fleet either. Rotation means a re-seal sweep over the affected rows, which is a thing the operator schedules rather than something that happens by itself.
I would rather have exactly one piece of state with this property, clearly named, than a system where "what happens if I lose X" has a different answer for six different X.
The secrets that are not in the environment at all
There is a second path for values that should not be in a machine's environment file, because an environment file is on the disk and the disk is in the snapshot.
Those are granted to the machine and fetched over the credential broker inside its own network namespace, opened with the fleet key at the moment of the request. They are in no snapshot and on no disk, which is what you want for the credential that would otherwise be sitting in a checkpoint you forked.
The split is worth being deliberate about. Configuration that is merely private can live sealed in a row and be delivered into the environment. A credential that must not be copied along with the machine should never be in the machine at all, and should be asked for.
Every note, or install the CLI and try the thing this one is about.