Why 'image: postgres:17' Built Fine and Never Started

Taking a flattened filesystem out of BuildKit means throwing away CMD, ENTRYPOINT, WORKDIR and ENV. Rebuilding Docker's merge rules by hand, and the bug that only showed up on stock images.

The report was one line: a compose file with image: postgres:17 in it built successfully and then the machine came up with nothing running.

No error. The build was green, the filesystem was there, the machine booted, and Postgres was not running. If you opened a shell you could start it by hand and it worked perfectly.

The cause is a consequence of a decision made much earlier, in a different part of the system, that I had not followed all the way to its end.

The decision: take the filesystem, not the image

Pilots does not run containers. It runs microVMs with a real root filesystem, so what a build needs to produce is an ext4 disk, not a layered image.

BuildKit's tar exporter is exactly right for that. It emits the flattened filesystem, so there is nothing to unpack and no layer machinery to reimplement. mke2fs -d takes the tarball and writes a disk.

What the tar exporter does not carry is image metadata. No CMD, no ENTRYPOINT, no WORKDIR, no ENV. Those live in the image config, which is a different artifact, and taking the filesystem alone means not taking it.

For an app with its own Dockerfile, nobody noticed, because we parse the Dockerfile anyway and the final stage says what to run. For image: postgres:17, where the "Dockerfile" is a single FROM line, the Dockerfile says nothing about what to run. Everything that matters is in the base image's config, and that was the half being dropped.

So the build produced a perfectly good filesystem with nothing to exec, and the agent, having nothing to exec, execed nothing.

The fix is a start spec written into the image

The build now writes a start spec into the filesystem at /etc/pilot-agent/start.json, assembled from two sources.

The first is the Dockerfile's final stage, parsed here.

The second is the base image's own config, which BuildKit's frontend has already resolved during the build. The trick is getting it out, and the obvious way does not work.

buildctl --metadata-file has a documented key for it, containerimage.config, and reading that key was the first version of this fix. It recovered nothing, on every host, silently. BuildKit 0.32 publishes containerimage.config.digest and never the config itself, so the code was asking for a key that was not there and falling back to the Dockerfile alone every single time. A fix that looks right, ships, and changes nothing is worse than no fix, because the bug is now believed to be closed.

So the build asks for a second output instead. An OCI layout directory beside the tarball, which is an index, a manifest and a config blob written to the spec, and the config is read out of that and the directory deleted as soon as it has been. The layout costs one uncompressed export that lives for the length of a build. What it buys is that the thing being read is an OCI artifact rather than an exporter's metadata key, so the next time a key moves between BuildKit versions, nothing here moves with it.

With both halves, a stock image builds and starts, and the compose file that started this can say image: postgres:17 and get a database. The metadata key is still read if the layout is missing, for a BuildKit that publishes the config inline. None currently does.

The merge rules are Docker's, field by field

Once you have two sources you need a rule for combining them, and the right rule is the one Docker already uses, because everyone's Dockerfile was written against those semantics.

The Dockerfile wins whatever it names.
The image fills in the blanks.
ENV merges key by key rather than wholesale.
PATH is filled last.
A Dockerfile ENTRYPOINT with no CMD discards the image's CMD.

That last one looks arbitrary until you think about what CMD is. It is the default arguments for the entrypoint. If a Dockerfile replaces the entrypoint, the image's arguments were written for a different program, and passing them to the new one is passing a Postgres flag to somebody's shell script. Inheriting them would be worse than dropping them, so Docker drops them, and so do we.

The spec records a from_dockerfile_only flag, and it is now meaningful rather than always true. False means the image's config was merged in. True means the build genuinely saw only the Dockerfile, which happens when the daemon published no config at all.

Fail at build time, where it is cheap

The other half of the fix is where the failure surfaces.

A start spec that still names nothing to run fails the build. Not the boot. The build.

That is a one-line policy with a large effect on what debugging feels like. At build time the error arrives in the log stream you are already watching, with the context of the thing you just changed, and an agent reading NDJSON can act on it immediately. At boot time the same fact arrives as a machine that came up and does not answer, with the health gate refusing to cut over, and the actual reason three layers down in a console log.

The general rule I took from this: when a pipeline has a stage that can prove an artifact is unusable, that stage should refuse it. Every stage after it will only report the symptom.

The part that generalises

This bug had a specific shape that I now go looking for. A system takes one of two artifacts because it only needs one, ships that way for months, and then meets an input where all of the information is in the artifact that was dropped.

The tar exporter decision was correct and is still correct. We do not want a layered image. The mistake was treating "we do not need the image" as "we do not need anything the image carries", and those turned out to be different statements separated by three months and one compose file.

Every note, or install the CLI and try the thing this one is about.