Close the Laptop, the Deploy Still Happens
If the client decides whether a successful build becomes a release, then a closed lid is a build that deployed nothing. On Pilots the deploy intent travels with the build and the host that ran it cuts the release.
The failure was quiet, which is why it took a while to see.
A build would succeed. The service would not have a new release. No error anywhere, because nothing had failed: the build really did succeed, and the thing that was supposed to happen next simply never ran.
What happened next lived on the client. The CLI watched the build stream, saw a success verdict, and then called the deploy endpoint. Which works, right up until the client is not there any more. A laptop that shut its lid. A terminal that was closed. A browser tab that navigated away. A CI job cancelled thirty seconds after it kicked off a build.
The build already outlived its connection, deliberately, in its own context. The verdict was the one piece still hanging off the other end of a socket.
The intent travels with the build
POST /v1/builds?deploy=<service> means "build this, and if it works, cut that service a release from it".
The host that ran the build cuts the release itself, on the verdict, exactly once, and the log's last line carries the release id.
Nothing on the other end of the connection decides anything. A client that hung up, a tab that was closed, a laptop that shut its lid, and two tabs watching one build all produce the same single rollout.
The refusal case moved too. If the image exists but the deploy is refused, most often by the health gate, that is a line in the build's log carrying the same error, code and next fields a direct deploy call would have answered with. The log is where a build's verdict is read, so the verdict belongs in the log rather than in an HTTP response nobody is holding.
One host cuts the release, and it is not always the one you called
There is a wrinkle that falls out of the single-writer rule. Only one host is allowed to write a service's rows, so a deploying build is forwarded to that host, the service's arbiter, and the build's log is held open across the rollout so that a follower sees the whole thing.
Which produces a second wrinkle. A build's log is host-local. It lives in the memory of the host that ran the build and 404s anywhere else. There is no forward for it, because nothing records which host holds a build, and adding a host_id column to the table that would need it is forbidden: that table has rows, and a column add on a populated Corrosion table backfills and gossips every row.
So after the forward, the host a client originally chose is routinely not the host with the log.
The answer is that the log is the fast witness and never the only one. The release the build cut is an ordinary replicated row, readable from every host in the fleet. A client that cannot reach the log polls the service instead. The dashboard's build log component does exactly that, and the fleet battery has a section that kills a build's client mid-build and then reads the release it cut from a host that did not build it.
Two witnesses, on purpose
I have come to think of this as a rule for any long-running operation behind an API.
There should be a fast witness that tells you what is happening right now, and it is allowed to be local, in-memory, lossy and unreplicated, because it is optimising for the common case where somebody is watching.
And there should be a durable witness that tells you what happened, and it must be replicated, readable from anywhere, and not require that anyone was watching.
The bug I started with was having only the fast one and building policy on it. The log stream was the only place the verdict existed, which made a watching client load-bearing for correctness.
The version that is wrong in the other direction is just as bad: making the durable record the only thing, and then the person running a deploy stares at a spinner because everything that would be interesting to watch was judged too ephemeral to keep.
A related read-time decision
While tidying this path I killed a column that had already been half-designed, and it is worth the paragraph because it is the same kind of thinking.
A service's depends_on is derived at read time, from the .internal addresses in both halves of its environment, and stored nowhere. A name is not a value. The dependency exists because an environment variable names another service, so reading it back out of the environment is reading the actual truth, and a stored copy would be a second answer that can disagree.
The schema rule pushed in the same direction, since a depends_on column would have been a column added to a table that has rows. Sometimes an invariant that exists for one reason saves you from a design mistake you were about to make for an unrelated one.
Every note, or install the CLI and try the thing this one is about.