A Memory Snapshot Has a CPU Vendor, and It Never Crosses
Firecracker memory snapshots carry raw CPUID, so an image photographed on Intel cannot be restored on AMD. Pilots runs a mixed fleet anyway, with a three-tier rescue that degrades to a cold boot instead of failing.
A Firecracker memory snapshot contains the guest's view of the CPU, and that view is raw CPUID. Restore it on a processor from the other vendor and the guest's instruction stream is being run by a chip it was never told about. CPU templates normalise within a vendor. They do not normalise across one.
This is not a limitation anyone can engineer away at our layer. It is the physical fact the fleet has to be designed around, and the design question is which of two bad answers to take.
Answer one: pick a vendor and buy only that. Every snapshot restores anywhere, the fleet is uniform, and you pay whatever that vendor costs at order time for ever.
Answer two: allow a mixed fleet and make the constraint explicit in the placement rules. More complexity, and the freedom to buy whichever box is cheapest when you need one.
We take the second, because this runs on auction bare metal and the whole cost story depends on buying the box that is cheap this month.
The disk half is vendor-free, which is what makes it survivable
The important structural detail is that a machine has two halves and only one of them is pinned.
The rootfs is an ext4 disk image. There is no CPUID in a filesystem. It is built by CI at a tag, pinned by a checksum file in the repo, and the bootstrap script refuses to ship a local golden rootfs that does not match the pin.
The memory template is never a file and is never shipped. It exists only as builds that a fleet host chunkified from its own boot and published through its own replica. No laptop can mint one. There is one per vendor pool, named golden-<vendor>, and each host records its own vendor in a row, read from /proc/cpuinfo and never inferred from whatever template happens to be lying around.
That split is the enforcement mechanism. It is not a check somebody remembered to write. The vendor-bound artifact is produced only by the machines that are allowed to produce it, and the vendor-free artifact is the one that gets built centrally and pinned.
Builder templates work the same way and for the same reason. A builder is idle-suspended between builds, so it has a memory image, so it is per pool: builder-<vendor>.
Three tiers of rescue
When a machine needs to come back, the ranking runs over a vendor-filtered candidate set, and it degrades in a stated order.
Tier 1 is a resume in place, while the owning host is alive. Nothing moved.
Tier 2 is a restore from object storage on another host of the same vendor pool. This is the ordinary cross-host case, and it is the one that keeps wake and rescue fast.
Tier 3 is what happens when no host of the machine's pool is alive. The hashed winner across the whole live fleet cold-boots the machine from its own disk instead of restoring the memory image, and records last_start = cold_boot so that the reason is legible afterwards.
Tier 3 keeps the id, the name, the URL, the volume, the agent token and every byte on disk. It loses the processes that were running and everything they held in memory. That is a real degradation, and it is a much better one than "this machine cannot be rescued".
The ranking for all three is the same hash over a filtered candidate set, so none of it needs a coordinator. It is the leaderless rescue with one extra filter.
What this means when you buy hosts
A mixed fleet wants at least two hosts per vendor for tier 2 to exist at all. With exactly one host per vendor, every rescue of that host's machines is a tier 3 cold boot, because there is no second host of the right pool to restore onto.
That is an operational rule with a sentence you can act on, which is the form these constraints should take. Not "prefer homogeneous hardware" but "two per vendor or your rescues cold-boot".
The two templates a fleet host may pin are T2CL on Intel and T2A on AMD, and that pairing is not arbitrary either. Firecracker designs those two for instruction-set parity, which is what makes a cold boot on the other vendor safe for the guest's instruction stream in the first place. A host whose CPU vendor or generation disagrees with its configured template is refused by the bootstrap, and hostd refuses to start on the same disagreement. Both checks exist because the failure they prevent is a guest executing instructions its kernel believes are unavailable, which is not a failure you want to discover from a customer.
The cold boot gets its own budget, deliberately
A cold boot is a kernel boot. It is fundamentally a different operation from a restore, and it takes about as long as Linux takes.
So it has its own budget rather than sharing the wake budget: under 5 seconds on dedicated hardware, and 30 seconds on a nested-virtualisation laptop node, against the engine's own nested-KVM kernel boot figure of about 25 seconds. It is asserted under its own flag, separately from the rest of the battery.
That separation is the point. If a cold boot shared the wake assertion, a regression in the restore path could hide behind the cold boot's looser number, and the one metric this product actually sells on would go quietly soft. Different operations get different budgets, and a budget that covers two operations covers neither.
The thing I would tell someone starting this
Write the constraint into the data model on day one.
Every memory image here belongs to the pool that photographed it, and that fact is recorded on the machine row, on the template row and on the host row. It is not derived at restore time, it is not inferred from the template the acting host holds, and it is not a convention. Once those three columns existed, every feature that touches restore got the constraint for free, including ones I had not thought of when I added them.
The alternative version of this system checks the vendor somewhere in the restore path, on the way in, in one function, and works fine until the second caller of that path forgets. The constraint is physical. The model should carry it.
Every note, or install the CLI and try the thing this one is about.