Cilium Native Routing (also known as Direct Routing or ENI mode in AWS) lets your pods communicate directly with other pods and services on the same network, bypassing the overhead of encapsulation.
Here’s how to set it up and what happens under the hood.
Let’s start with a minimal cluster and see how it works. Imagine two nodes, node-1 and node-2, each with a single pod.
Node 1:
- IP:
192.168.1.10 - Pod
pod-a(IP:10.0.0.1)
Node 2:
- IP:
192.168.1.11 - Pod
pod-b(IP:10.0.0.2)
Without native routing, traffic from pod-a to pod-b would be encapsulated (e.g., VXLAN or Geneve), sent to node-2, decapsulated, and then delivered to pod-b. This adds latency and CPU cycles.
With native routing, node-1 knows that 10.0.0.2 is reachable via node-2’s IP address (192.168.1.11). It directly routes the packet to node-2, which then delivers it to pod-b without any encapsulation.
To enable this, you need to configure Cilium to use the node’s IP address as the gateway for pod traffic and ensure the underlying network infrastructure can route traffic between node IPs and pod CIDRs.
Configuration:
-
Enable Direct Routing in Cilium: When installing Cilium, you’ll use specific Helm values. The key is setting
enable-ipv4-native-routingtotrue.helm install cilium cilium/cilium --version <cilium-version> \ --namespace kube-system \ --set ipam.mode=kubernetes \ --set ipv4.enabled=true \ --set tunnel=disabled \ --set autoDirectNodeRoutes=true \ --set enable-ipv4-native-routing=true \ --set cluster-pool-ipv4-cidr=<your-cluster-cidr>tunnel=disabled: This explicitly disables IP-in-IP or VXLAN tunneling.autoDirectNodeRoutes=true: This tells Cilium to automatically configure routes on the nodes. For each node, it adds a route for the node’s pod CIDR pointing to the node’s IP address. This is crucial for direct routing.enable-ipv4-native-routing=true: This is the primary flag for enabling direct routing mode.cluster-pool-ipv4-cidr: This should be the IP address range allocated for your pods (e.g.,10.0.0.0/8).
-
Network Infrastructure: The most critical part outside of Cilium configuration is ensuring your network can handle the routing.
- Cloud Providers (AWS ENI Mode): If you’re on AWS, you’d typically use
aws-eniIPAM mode. Cilium will manage Elastic Network Interfaces (ENIs) and their associated IP addresses. The VPC routing tables must be configured to allow traffic from your VPC’s subnet CIDR to reach the pod CIDR, and vice-versa. Cilium, whenautoDirectNodeRoutes=true, will add routes to the node’s local routing table, but the VPC must facilitate inter-node communication. - On-Premises/Bare Metal: You need to ensure that your network devices (routers, switches) know how to route traffic between the IP addresses of your nodes and that they can reach the pod CIDR. This often involves adding static routes on your network infrastructure or using a dynamic routing protocol. For example, if your nodes have IPs
192.168.1.10and192.168.1.11, and your pod CIDR is10.0.0.0/8, your network must understand that10.0.0.0/8traffic destined for192.168.1.11should be sent to that node’s IP.
- Cloud Providers (AWS ENI Mode): If you’re on AWS, you’d typically use
How it Works Internally:
When enable-ipv4-native-routing=true and autoDirectNodeRoutes=true, Cilium does two main things:
-
Disables Encapsulation: It configures the networking stack to not use tunnels. Packets originating from pods are treated as regular IP packets.
-
Configures Node Routes: For each node in the cluster, Cilium adds a route to the node’s kernel routing table. This route states that the node’s assigned pod CIDR (e.g.,
10.0.0.0/24fornode-1) is directly reachable via the node’s primary IP address (e.g.,192.168.1.10).You can see these routes on a node using
ip route show. You’ll see entries like:10.0.0.0/24 via 192.168.1.10 dev eth0 # Example for node-1 10.0.1.0/24 via 192.168.1.11 dev eth0 # Example for node-2(The exact device
eth0will vary).When
pod-a(onnode-1) wants to send a packet topod-b(onnode-2), the kernel onnode-1looks at the destination IP (10.0.0.2). It consults its routing table and finds the route for10.0.0.0/24(or the broader pod CIDR). This route points tonode-2’s IP (192.168.1.11). The packet is then sent directly fromnode-1tonode-2over the underlying network infrastructure.node-2receives the packet and, knowing it’s for a local pod, delivers it without any further processing or encapsulation.
Verification:
-
Check Cilium Status:
cilium status --verboseLook for
Native Routing: Enabledand ensure no tunneling protocols are listed as active. -
Check Node Routes: SSH into your nodes and run:
ip route showYou should see routes for each node’s pod CIDR pointing to the respective node IPs.
-
Packet Capture (Optional): On
node-1, you can capture traffic on the interface connecting to your network (e.g.,eth0) whilepod-apingspod-b.sudo tcpdump -i eth0 host 10.0.0.2 -nYou should see IP packets with the source IP
10.0.0.1and destination10.0.0.2going out directly to192.168.1.11. No VXLAN or other tunnel headers should be present.
The one thing most people don’t realize is that autoDirectNodeRoutes=true doesn’t just add routes on the node itself; it also ensures that Cilium’s CNI plugin configures the local IP address of the node as the source IP for these routes. This is crucial because without it, the node might try to route traffic using a different interface’s IP or even try to encapsulate it if tunneling were still implicitly enabled.
The next thing you’ll likely encounter is configuring Network Policies to control this direct traffic flow between pods.