Every Guest Has the Same IP, So How Do Two of Them Talk?

Every machine on Pilots sees exactly the same network. Its interface is 169.254.0.21/30 and its gateway is 169.254.0.22. Every single one, on every host.

That is not laziness. It is the reason a snapshot restores anywhere. A memory image contains the guest's networking state, so if the address were host-specific, the snapshot would be host-specific, and the whole restore-on-any-host property would be gone.

Which leaves an obvious problem. If your web machine and your database machine both believe they are 169.254.0.21, how does one connect to the other?

Nothing host-specific may be inside the guest

Every answer below comes from one principle: everything that varies by host lives where a restore rebuilds it, which is the host side of the boundary.

Names are <machine>.internal, answered by a DNS responder hostd binds on 169.254.0.22:53 inside each machine's own network namespace. The answers come from the local replica, scoped to the app the asking machine belongs to and filtered to healthy machines. Everything else is forwarded upstream.

The guest's /etc/resolv.conf names only 169.254.0.22, and it is written at rootfs build time. A constant, so it is snapshot-safe by construction rather than by a fixup that runs at restore.

Addresses are derived, never allocated

A cluster-wide subnet allocator would be a control plane, which is the rule this system does not break. So each host derives its addressing from its own WireGuard public key, using two domain-separated prefixes:

host     fd cc + pubkey[0:14]     /128    hostd listens here
machine  fd cd + pubkey[0:12]     /112    machines at <prefix>::1 .. ::400

The machine prefix spends 16 bits on the slot index, leaving 96 bits of derivation entropy. A birthday collision sits past 2^48 hosts against a fleet of tens, and the slot index is the one the network namespace pool already allocates, so nothing new is being allocated at all.

Two prefixes rather than one wide one is the part I would keep if I could keep only one decision. It makes the tenant boundary structural: guests may only ever reach fdcd::/16, and hostd only ever listens under fdcc::. One static rule enforces that, with nothing to reconcile and nothing to get wrong on a host added next year.

The alternative is an enumerated deny: block the host's own address and the host service ports inside every machine prefix. That has to stay correct on every host for ever, and a rule like that is only ever one new service port away from being wrong.

Guests speak IPv4 and the mesh is IPv6

The two are bridged at the host rather than wished away. Every guest additionally holds a constant fdee::21, identical on every guest for the same reason .21 is, and equally snapshot-safe.

DNS answers with the peer's derived fdcd address. The namespace then translates both ways: source-rewrite outbound fdee::21 to <prefix>::<slot>, destination-rewrite inbound back to fdee::21. The root namespace routes and filters and never translates. Cross-host, the packet takes the mesh to the owning host and is translated in the target namespace there. Same path either way.

Why translating in the root namespace cannot work

This is the part worth keeping, because the arrangement looks natural until you try it.

After translation the address is fdee::21, which is the same address in all 1024 of a host's namespaces. Netfilter makes its routing decision after the prerouting destination rewrite, and it does not carry the ingress interface into that decision. So both the inbound direction and the reply to an outbound flow arrive at a thousand candidate virtual ethernet interfaces with nothing to choose between them.

Recovering the answer needs a connection mark and a policy-routing table per namespace. That is a second addressing scheme layered on the derived one, reconciled on every machine that moves, which on this platform is every rescue.

Translating one hop earlier means the packet already carries an address that is unique on the host by the time anything has to route it. The rule is small and general: do the translation at the boundary where the identity still exists, not after you have erased it.

Isolation belongs at the root-namespace hop

Every guest sources from the same 169.254.0.21, so classification by source address is impossible by construction. Classification is by ingress virtual interface, which maps to a machine, which maps to an app.

Doing that in the root namespace costs one rule set per host, updated once per fleet change. Doing it inside each namespace would cost a set of every peer address in the app, in up to 1024 namespaces per host, re-reconciled on every fleet change and churning hardest during a rescue, which is exactly when the host is busiest.

The same rules drop, per interface, any IPv6 whose source is not that slot's own machine address. Against the ingress match that looks redundant. It is not: without it a guest can put a peer's address in the packets it sends. The reply goes to the real owner, so it steals no traffic, but the receiving machine's filter sees a connection opening from inside its own app and accepts it. The ingress interface is the host's knowledge. A source address is the guest's.

Three landmines, all about addresses moving

Near-zero DNS TTL, always. A rescued or cold-booted machine lands on a new host with a new slot, so its address changes, and a guest holding a 300 second answer talks to nothing. A wake on the owning host is normally not a move, because the replica takes back the index it kept while suspended. Two things still move an address: a rescue, and a wake whose kept index was handed away while hostd was down, since the pool is rebuilt from running machines only.

A low TTL does not save an established connection. A pool holding an open socket to a rescued machine's old address simply breaks. That is what every failover does, and it means .internal clients need reconnect logic. "No human action" covers the platform, not your application's connection pool.

Key rotation is a readdressing event. Machine addresses derive from the host's key, so rotating it moves every machine that host runs. Drain the host first, or accept a connection reset for all of them.

And one that cost a genuinely confusing afternoon: a tap device that came back from a rebuild with a fresh MAC address left the restored guest's neighbour table pointing at a MAC nothing had. Its IPv6 replies went nowhere for the fifteen seconds unreachability detection takes to give up on unicast probes, which reads from the outside as a .internal name that resolved and did not answer. Every namespace's tap now carries one constant MAC, for the same reason every guest carries one constant IP.

Every note, or install the CLI and try the thing this one is about.