The Signal That Killed One Restore in Four
NBD_DO_IT parks a thread for the life of the device, and any signal delivered to that thread tears the device down. Go's own scheduler preemption was doing it. The symptom pointed nowhere near the cause.
Roughly one restore in four failed. Not with an error from the thing that was broken, which would have been too easy. With a timeout, from a caller waiting for a block device to report its size, against a device that looked completely free.
The sequence in the logs read like this. Attach the network block device. The kernel logs a capacity change, so the device came up. The size then returns to zero. The caller waits for a non-zero size, does not get one, and fails. Inspect the device afterwards and /sys/block/nbdN/pid is empty, which means no process owns it, which means it is available, which means nothing is wrong with it.
Everything in that paragraph is true and none of it points at the cause.
What NBD_DO_IT actually does
A network block device is a kernel device whose blocks are served by a userspace process over a socket. You hand the kernel a socket, then you issue the NBD_DO_IT ioctl, and that call does not return. It parks in wait_event_interruptible for the entire life of the device, which is how the kernel keeps a thread available to drive the queue.
Read that word again: interruptible. If any signal is delivered to the thread sitting in that call, the kernel wakes it up, runs sock_shutdown and nbd_clear_que, and tears the device down.
Now consider what a Go program does to its own threads. The runtime preempts goroutines asynchronously by sending SIGURG to the thread running them. It is a completely ordinary part of how the scheduler works, it happens constantly, and it is invisible unless something you are doing cares about signals.
The handler cared. The kernel got a signal on the parked thread, shut the device down, and the size went back to zero. One in four was just how often the scheduler happened to preempt that particular thread during the window.
Three lines of fix
runtime.LockOSThread, so the goroutine owns a thread nothing else will be scheduled on. Mask signals on that thread for the duration of the call. Issue NBD_DO_IT from there.
That is the whole repair, and it is the kind of fix that is obvious in hindsight and unreachable from the symptom. Nothing in "a device reports no owner and a caller timed out waiting for a size" says "your language runtime's preemption signal is tearing down a kernel device". I found it by reading the kernel's nbd driver, not by reading our code, which is the general lesson I keep relearning: when the symptom makes no sense, the answer is usually in the layer you have been treating as a black box.
The same code path had three more of these
The block device layer turned out to be the densest concentration of this kind of thing in the whole engine. Since they cost real days, they are written down in the architecture doc, and here.
A failed NBD_DO_IT has to be surfaced, not collected. The first version gathered errors from the setup goroutines and reported them at the end of the attach. Which meant a failure that happened inside the parked call had nothing anywhere naming it, and the only evidence was the sizing timeout again. An error that reaches a log five layers from its cause is barely better than no error.
Issue NBD_DISCONNECT before killing a handler, and issue it from the parent. A handler blocked in NBD_DO_IT never reaches its own cleanup code, because it is not running any code. Kill it without disconnecting first and Firecracker blocks in uninterruptible sleep with /dev/nbdN dead until the host reboots. A stuck D-state process on a multi-tenant host is not a bug you get to fix at your leisure.
Pick free devices with /sys/block/nbdN/pid, not by probing. The obvious check is to open the device and ask its size with blockdev. That hangs on a half-attached device, so the probe you wrote to avoid a problem becomes the problem. Reading the pid file is a read of a text file that either has a number in it or does not.
Hand devices out round-robin, not lowest-free. The kernel publishes a device's size and clears it asynchronously, and those two operations are not ordered against each other. Lowest-free means you immediately reuse the device you just released, in exactly the window where its state is still settling. Round-robin means by the time you come back to a device, the kernel has had a full cycle to finish putting it away. This is not a correctness fix so much as a way of not standing in the one place where the race is reachable.
Why any of this is in our code at all
A reasonable question at this point is why we are issuing kernel ioctls from Go rather than using something that already exists.
The answer is what the block layer is for. A machine's root disk on Pilots is not a file on the host. It is a read-through overlay: a template shared by every machine on the host underneath, the machine's own written blocks on top, and object storage behind both. A host that has never held the template attaches the device immediately and serves reads from the bucket, range by range, while a background prefault pulls the rest.
There is no existing block device that does that, and the shape of it is the entire reason a machine's disk costs nothing when it changes nothing. So the handler is ours.
It is also, explicitly, a port rather than a rewrite. The AGENTS.md rule in the repo says that in those words for both this handler and the userfaultfd one, because they encode kernel ABI details that are expensive to rediscover, and this post is four paragraphs of evidence for why that rule exists. Every one of these cost a day or more, and not one of them is discoverable from our own logs.
Every note, or install the CLI and try the thing this one is about.