Cilium Cluster Mesh lets you manage multiple Kubernetes clusters as a single, unified network, but its real magic is how it makes them forget they’re separate entities.
Imagine you have two Kubernetes clusters, cluster-a and cluster-b. Normally, pods in cluster-a can’t talk to pods in cluster-b without some explicit network plumbing. Cilium Cluster Mesh makes this seamless.
Let’s see it in action. We’ll assume you have two clusters, cluster-a and cluster-b, both running Cilium with the Cluster Mesh feature enabled.
First, we need to establish trust between the clusters. This is done using shared secrets. On cluster-a:
kubectl get secret -n kube-system cluster-mesh-config -o jsonpath='{.data.cluster\.id}' | base64 --decode > cluster-a.id
kubectl get secret -n kube-system cluster-mesh-config -o jsonpath='{.data.cluster\.secret}' | base64 --decode > cluster-a.secret
Then, on cluster-b, you’d do the same, creating cluster-b.id and cluster-b.secret.
Now, we’ll create a ClusterMesh resource in each cluster. This tells Cilium about the other cluster and how to connect.
On cluster-a:
apiVersion: cilium.io/v2alpha1
kind: ClusterMesh
metadata:
name: my-mesh
spec:
clusters:
- name: cluster-b
secret:
name: cluster-b-secret # This secret will be created on cluster-a
namespace: kube-system
And on cluster-b:
apiVersion: cilium.io/v2alpha1
kind: ClusterMesh
metadata:
name: my-mesh
spec:
clusters:
- name: cluster-a
secret:
name: cluster-a-secret # This secret will be created on cluster-b
namespace: kube-system
Before applying these, you need to exchange the secrets. On cluster-a, create a secret that cluster-b can use:
kubectl create secret generic cluster-a-secret \
--from-file=cluster.id=cluster-a.id \
--from-file=cluster.secret=cluster-a.secret \
-n kube-system
And on cluster-b, create a secret that cluster-a can use:
kubectl create secret generic cluster-b-secret \
--from-file=cluster.id=cluster-b.id \
--from-file=cluster.secret=cluster-b.secret \
-n kube-system
Now apply the ClusterMesh resources.
With this set up, if you deploy a service named my-app in cluster-a and another service named my-app in cluster-b, pods in cluster-a can access my-app.my-namespace.svc.cluster.local and reach the pods in cluster-b directly, and vice-versa. Cilium handles the routing and encryption (if configured) transparently.
The core problem Cluster Mesh solves is the isolation of Kubernetes networks. Each cluster typically has its own IP address space and network policies, making inter-cluster communication a complex, manual undertaking. Cluster Mesh collapses these boundaries by creating a unified network fabric. It achieves this by:
- CEP (Cluster-External-Service) Discovery: Cilium discovers services in remote clusters. When a service is exposed cluster-externally, Cilium registers it as an endpoint for pods in other connected clusters.
- Global Service Discovery: DNS resolution is extended. When you query
my-app.my-namespace.svc.cluster.local, Cilium’s DNS proxy can resolve to endpoints in any cluster within the mesh, not just the local one. - IP Address Management (IPAM): Cilium ensures that the IP address ranges used by pods and services in different clusters do not overlap, preventing routing conflicts. This is configured during Cilium installation.
- Encryption: Communication between pods across clusters can be automatically encrypted using WireGuard, ensuring secure data transfer.
The secret exchange mechanism isn’t just for authentication; it’s how each cluster learns the identity and shared secret to establish secure, authenticated communication channels with its peers. This shared secret is then used to derive keys for encrypting traffic between clusters.
What most people don’t realize is that the ClusterMesh resource itself doesn’t directly contain the connection details. Instead, it references Secret objects that hold the cluster.id and cluster.secret files. These secrets are generated by Cilium during its initial setup on each cluster and are crucial for establishing mutual TLS authentication between the Cilium agents (cilium-agent) running in different clusters. The ClusterMesh resource acts as a declarative way to tell Cilium, "Here’s which other clusters I want to be part of this mesh, and here’s how you can find their connection credentials."
Once you have multiple clusters successfully meshed, the next logical step is to explore advanced routing policies and service discovery patterns, like prioritizing local endpoints or implementing cross-cluster ingress.