Why every deploy should produce an immutable image tag

A container image tag looks like a version. It is not. It is a mutable pointer, and any tag can be moved to a different image at any time. This is the source of a specific class of bug that is unusually unpleasant to diagnose.

The failure

Three replicas run an app deployed as myapp:latest. One is rescheduled onto a node that has not pulled that tag before, so it fetches the current latest. Meanwhile the tag has moved.

Now two replicas run one build and the third runs another. There was no deploy. Nothing in any log says a version changed. Requests fail intermittently, in proportion to how often the load balancer picks the odd one out, which is the sort of ratio that makes people doubt their own reproduction steps.

The same mechanism produces a subtler version: a rollback that does not roll back. You redeploy myapp:v2.1 expecting the build from last week, and get whatever v2.1 points at now, because someone rebuilt and re-pushed under the same tag.

The fix

Make every build produce a tag that will never refer to anything else. The usual form includes the commit: myapp:main-a1b2c3d. Two builds of different commits cannot collide, and a tag you deployed last month still means what it meant.

Underneath tags there is a stronger identifier still. Every image has a digest, a hash of its content, written as sha256:.... A digest cannot be moved by anyone, because changing the content changes the digest. Deploying by digest is the strongest form of this guarantee.

The cost is honest and small. You need a build that generates the tag, a deploy path that consumes it rather than a fixed string, and a registry that accumulates more tags than before, which is a retention policy you now have to have.

What it buys beyond correctness

• Rollback becomes exact, because the earlier tag still identifies the earlier bytes.
• The running version is a fact, readable from the workload rather than inferred from deploy history.
• Incidents get shorter, because "what is actually running" stops being a question.
• Builds become cacheable, since the same commit produces the same tag and does not need rebuilding.

How Kaja handles it

Builds from a connected repository produce a tag derived from the branch and the commit, so every deploy is addressable after the fact. Each one is also recorded as a numbered revision holding the image reference and the configuration applied with it, which is what makes rolling back a re-application of known state rather than a rebuild.

The deployments timeline shows the image reference on each entry alongside the commit it came from, so the question "which build is live" has a visible answer rather than a procedure. If you build images in your own CI instead, the same applies as long as your pipeline tags by commit, and you can point Kaja at those images directly.

We wrote about the rollback half of this in what "roll back" should mean.