Hold the Request, Don't Show a Waiting Page
Scale-to-zero usually means a visitor sees a loading screen while something starts. On Pilots the connection is held open while the microVM restores, so the first request is just a slow request.
Scale to zero has an ugly moment in it. Nothing is running, a request arrives, and something has to start before there is anything to answer with. Every platform that scales to zero has to decide what the visitor sees during those seconds.
The common answers are a holding page that refreshes, a 503 with a retry header, or a redirect that lands somewhere useful once the thing is up. All three make the visitor a participant in your capacity planning.
Pilots holds the connection. The request arrives at the router, the router sees the machine is suspended and that auto-start is on, and it does not answer. It restores the machine, proxies the request, and the response goes back on the same connection the visitor opened. From the client's point of view it was one request that took a while.
On dedicated hardware that while is under 200 ms for a wake, which is inside the range that reads as "the server thought about it" rather than as an outage.
What suspended actually means here
This only works because suspend is complete rather than partial.
When a machine suspends, the Firecracker process is killed. The memory is gone from the host. The network slot is released. There is no reserved memory, no veth, no parked process, and no per-machine object waiting for it to come back. What remains is rows in a replicated database and its snapshot in object storage.
That is a stronger claim than it sounds, because the easy version of scale-to-zero keeps something warm per sleeping workload, and then a thousand sleeping workloads cost real memory. Here the whole host has one dummy interface standing in for the entire wake address block, and that is the extent of what sleeping machines cost a host.
So the cost of a suspended machine is the storage its snapshot occupies. Nothing else.
It also means the wake does not have to happen on the host that suspended it. Nothing about the machine is on that host any more, so any host of the right vendor pool can restore it from object storage. Cross-host is the ordinary case rather than the exception.
Deciding when to suspend is the harder half
Waking is mechanical. Knowing when to sleep is where the judgement is, and the idle monitor asks three questions in order.
Is the wall clock idle? The idle_timeout knob defaults to 60 seconds, tops out at an hour, and is per machine.
Is concurrency zero? In-flight requests against the soft limit. A machine with a long request still running is not idle regardless of what the timer says.
Both of those have to agree. And critically, exec and WebSocket activity count as activity. An agent running a twenty-minute build over an exec stream, with no HTTP traffic at all, is not idle. That sounds obvious and it is exactly the bug that makes agent sandboxes miserable on platforms that only count HTTP.
Is a console session still running a command? This is the last question, asked only for a machine the first two already agreed on. The host asks the guest agent, which answers from its own process tree: is there any process besides the session leader with the same session id. The host cannot answer this itself, because a client that detached took hostd's only view of that session with it.
Two probes, two opposite failure directions
The console probe fails open. If the guest does not answer, the machine suspends.
The guest-to-guest traffic signal fails safe. If conntrack says an established connection is open from a running peer, the machine stays up.
Those are deliberately opposite, and the reason is what each mistake costs.
A suspend is reversible. It is a freeze that resumes on the next touch, so suspending a machine that was busy costs a wake, which is 200 ms. Failing open is choosing the recoverable mistake.
Holding a machine up that should have slept costs money for as long as nobody notices. But dropping a peer's established connection breaks an application that is actively using it, which is not recoverable by anything the platform does. So that one fails safe.
Whenever there is a probe whose answer might be missing, it is worth asking explicitly which mistake you would rather make, rather than letting the default fall out of how the code happens to be written.
Sandboxes and services are governed by different things
The idle monitor owns sandboxes. A machine with a release, which means a rollout's replica or a promoted sandbox, belongs to the autoscaler instead.
The split matters because the questions differ. A sandbox is one machine and the question is "is anyone using this". A service has a floor, replicas, and a scale-down window, and the question is "does this service still need this many".
The host that holds a replica gives it back when its own in-flight count, its held sessions and the row's last-activity all say idle for the scale-down window, and the floor allows it. Adding capacity is the arbiter's job alone. Every host ranks the same replicated rows the same way, so exactly one host ever concludes that it is the one to act, which is the leaderless pattern again rather than a lock.
The default floor is zero on both faces. A promoted service still suspends when nobody is using it, because promote changes what governs the machine and not the knobs it had.
The thing I actually wanted
I wanted a prototype I could leave running for three months and pay nothing for, with a URL I could send to somebody, that answers when they click it.
Not a container that gets evicted. Not a free tier that sleeps and shows a branded loading page. A machine with an address, asleep, that wakes because someone knocked on the door, fast enough that they never learn it was asleep.
The number that makes it work is the wake, and the reason the wake is that fast is restore rather than boot with lazy memory behind it. The held connection is just the piece of plumbing that turns the fast wake into something a visitor never has to know about.
Every note, or install the CLI and try the thing this one is about.