Multi-cluster deployment, without inheriting multi-cluster complexity

Almost nobody sets out to run several Kubernetes clusters. You end up there. A second region, an acquisition, a customer who requires their workload on their hardware, a development cluster that quietly became load-bearing, a compliance boundary that could not be a namespace.

The awkward part is that a lot of tooling assumes one cluster, and the assumption is invisible until the day it is wrong.

What actually gets harder

Knowing where anything is. With one cluster, "is the API deployed?" is a question with an answer. With five, it is a question per cluster, and the honest answer is often "I would have to look."

Context switching. Cluster context lives in a file on each engineer's laptop. Every destructive command is now preceded by checking which cluster you are pointed at, and the checking is manual, which means it is occasionally skipped.

Configuration drift. The ingress controller was upgraded on three of them. Two clusters have a monitoring stack from a different era. Nobody did anything wrong; they were each fixed on the day they broke.

Access control. RBAC is per cluster, so "this contractor can view staging and touch nothing else" is a policy you now apply five times and audit five times.

Reachability. Some of these clusters are not on the public internet, and the ones that are not are usually the ones that matter most.

What does not have to get harder

Most of that difficulty is not inherent to running several clusters. It comes from every cluster being addressed individually, by tools that hold no shared model of what should be running.

The alternative is to make the cluster an attribute rather than a destination. You work in terms of a project and its apps; which cluster it lands on is a property of the project, set when it is created. The list of what exists is one list. Roles are defined once for the organisation and mean the same thing everywhere.

How Kaja does it

An organisation holds any number of clusters, and one console covers all of them. Each cluster runs an agent that dials out, so adding one is an install command rather than a networking project, and clusters with no public address work the same as clusters that have one.

Projects belong to a cluster, and the console works in projects. Roles and the audit log are organisation-wide, so an access decision is made once. When you want the same shape somewhere else, a project can be saved as a template and duplicated, including onto a different cluster — which is the difference between "recreate staging in the new region" being an afternoon and being a fortnight.

Worth being clear about one limit: images built inside one cluster are in that cluster's registry, so a duplicate onto a different cluster rebuilds rather than copying bytes across. That is a consequence of builds running on your own hardware, which is also the reason they are not billed by the minute.

More in connecting clusters and templates and duplication.