Skip to content
pilots
Dashboard
All pages

Errors

Every failure answers in the same shape, and the useful field is the one naming what to do next. Branch on the code, never on the wording: the sentence is written for a person and it will be reworded.

{
  "error": "a sentence for a person",
  "code": "health_gate_failed",
  "next": "the one thing to do about it",
  "details": { },
  "request_id": "the name the host logged this request under"
}

code is a closed list. next names the command or the call to make now. details is typed per code and absent otherwise. request_id is the same value as the X-Request-Id header on every answer, and it is what to quote when you report one. The CLI prints the error and then next on its own line, and --json prints the server's body unchanged.

Every code

Code Status What it means
bad_request 400 The request is malformed or missing a field. The answer names which.
unauthorized 401 No key, or a key that does not resolve. Sign in, or set the key variable.
scope_required 403 The key resolves but lacks the scope, or is held to a name prefix and reached past it. Mint one that has the scope, or act on what the prefix covers.
self_only 403 A machine’s own token reached past the machine it was minted for. Act on that machine, or use an API key for anything wider.
repo_not_connected 403 This organization has no claim on the repository named. Connect it with an admin-scoped key, or send a tar of the directory instead.
billing_hold 402 The team has used its free allowance for this month, or its bill reached the budget it set. Its machines are asleep and nothing was deleted. Run pilot billing to add a card or change the budget, and the machines wake as usual once the hold lifts.
not_found 404 No such object, or this key belongs to another organization.
app_required 400 A service with no app was given an environment that names a .internal address. Those names resolve only inside one app, so set app to the app of the service it connects to.
conflict 409 An operation is already running on it. Wait for it to finish.
rollout_in_progress 409 Another deploy, rollback or resize of this service is still rolling. Wait for it: pilot service deploys <service> shows it.
release_unrestorable 409 A rollback or redeploy named a version whose image is no longer stored. Deploy the service from source with pilot deploy.
checkpoint_unrestorable 409 A fork or restore named a checkpoint whose saved state is no longer fully stored. Take a new checkpoint, or fork the machine itself.
state_lost 409 A sleeping machine was woken whose saved disk is no longer fully stored, so it cannot come back. Create it again, or redeploy its service.
volume_in_use 409 A volume is attached or mounted elsewhere. The answer names what holds it.
payload_too_large 400 A field is too big to live in a replicated row. Put the value in a volume or in object storage and reference it.
no_capacity 507 The request was fine and the fleet has no room for the machine right now. This is the fleet's capacity, never a limit on your account: no team is held to a ceiling. Retry shortly or ask for a smaller one.
busy 503 A host had no room for the create or fork this moment, held it in its line for up to 20 s, and found none. Wait the time in the Retry-After header and send the same request again. The CLI does this by itself.
not_configured 400, 501, 503 This host or fleet was built without the piece. The answer names it.
not_implemented 501 The route is not built yet.
unavailable 503 Something this needs did not answer: the host that writes this object, or a service's volume. When it is the volume, the deploy did not start and the service stays on its release. Retry in a minute.
internal 500 The host broke. Retry once, and say so if it repeats.
plan_unsupported 400 The compose file asks for something pilots does not do.
compose_invalid 400 The compose file does not parse. The message names the file.
unknown_framework 400 No compose file, no Dockerfile, no recipe.
plan_multi_service 400 A push tried to deploy a repository holding more than one app.
build_failed on the log line The build stopped. The line carrying the failure says where.
health_gate_failed 422 The replica never answered its health check.

The ones that carry details worth reading

health_gate_failed carries the service, the replica, the release, the grace it was given, and the last thing the probe saw: a status and a body when the replica answered, and an error when it did not. It carries NO address, because the probe target is the host's own view of the replica and is not reachable from where you are reading the error. A refused connection there means one of two things and almost nothing else: the app is listening on the wrong port, or it bound the loopback address instead of all interfaces.

It also carries console: the replica's last 40 console lines, with every value of the service's environment and anything shaped like a credential replaced by [redacted]. That is usually the answer outright, a missing module or a crash on boot. The replica is parked rather than destroyed, so pilot machine logs on it still has the whole console. A build that was asked to deploy ends its log on a line carrying the same code, next and details, replica and console included.

scope_required for a prefix carries name_prefix, what the key is held to, and name, the thing it reached for. It is a refusal and never a not_found, on purpose: a client that reads "not found" as "the machine is gone" would make a fresh one over a refusal.

unknown_framework carries the directory, what was looked for, the listing, the manifests, any workspaces, and the rules a Dockerfile must obey. That is enough to write one without opening the repository again (deploy).

busy carries retry_after_ms, position and queue_length. Each host keeps its own line for starts that do not fit yet, because machines sleep seconds after going quiet and free their memory. The position is your place in the line of the host that answered, and it is approximate: there is no line across the fleet, and a retry may start on another host. The SDKs raise it as a typed error with the wait and the place, and pilot waits and tries again for up to three minutes, printing the wait and the place while it does.

What to do

You got Do
Any error Read next and do that.
health_gate_failed Diagnose the replica named in the answer, then fix the app. Do not redeploy first.
build_failed Read the log line carrying the failure, fix it, build again.
unknown_framework Write the Dockerfile from the details, build with it, and deploy that build under a name.
internal Retry once. If it repeats, say so rather than working around it.
busy Wait details.retry_after_ms and send the same request again.
A 503 about a machine whose owner is gone Retry. The host that would bring it back is still joining the fleet and claims nothing until its replica has caught up. A host says so about itself on its health route, and it clears itself within seconds.

Three habits to avoid

  • Parsing the sentence. It is for a person and it will be reworded.
  • Retrying a failed health gate by deploying again. Nothing changed, so nothing will.
  • Treating a not_found as proof the object does not exist. A key scoped to another organization gets the same answer, on purpose.