XDP bypasses the Linux kernel’s network stack entirely, allowing you to run eBPF programs directly on incoming network packets at the earliest possible moment. This means packets can be dropped, forwarded, or even modified before they ever hit the kernel’s main processing path, dramatically reducing CPU overhead and latency.
Let’s see XDP in action with Cilium. Imagine we have a simple load balancer forwarding traffic to a few backend pods.
# First, ensure you have Cilium installed with the appropriate XDP flags.
# This is typically done during Helm installation:
helm install cilium cilium/cilium --version <cilium-version> \
--namespace kube-system \
--set xdp.enabled=true \
--set xdp.mode=xdp-native \
--set ipam.mode=kubernetes
With XDP enabled and set to xdp-native mode, Cilium’s eBPF programs are loaded directly into the network interface’s XDP hook. When a packet arrives at the NIC, the XDP program intercepts it. If it’s destined for a service managed by Cilium, the XDP program consults Cilium’s internal maps (like the lb4: map for IPv4 load balancing) to determine the next hop.
Here’s a simplified view of what happens:
- Packet Arrives at NIC: The network interface receives an incoming packet.
- XDP Hook: The packet is passed to the XDP program loaded by Cilium.
- Cilium XDP Program Execution:
- The program checks if the packet’s destination IP and port match a Cilium-managed service.
- If it matches, it looks up the service in the
lb4:orlb6:eBPF maps. - Based on the load balancing algorithm (e.g., round-robin), it selects a backend pod IP and port.
- Crucially, instead of sending the packet up to the kernel’s IP layer and then to kube-proxy or Cilium’s agent, the XDP program can directly rewrite the destination MAC and IP addresses (if necessary) and steer the packet directly to the intended backend pod’s veth pair or another network interface.
- If the packet is not for a Cilium-managed service, it’s typically passed through to the kernel (
XDP_PASS). - If the packet is invalid or should be dropped, it’s dropped (
XDP_DROP).
This direct manipulation at the XDP layer is what yields the significant performance gains. No context switches into the kernel’s full network stack are needed for accelerated traffic.
The primary problem XDP acceleration solves is the bottleneck created by the traditional Linux network stack. For high-throughput networking (e.g., DPDK applications, high-traffic load balancers, or network functions), the CPU cycles spent processing each packet through the kernel can become a limiting factor. XDP allows you to offload this processing to the network card’s driver or a specialized eBPF engine, freeing up CPU cores for application workloads.
The exact levers you control with Cilium’s XDP integration are:
xdp.enabled: Toggles XDP processing on/off.xdp.mode:xdp-native: The most performant mode, loading eBPF directly into the NIC driver’s XDP hook. Requires NIC driver support.xdp-generic: A fallback mode that uses a generic XDP hook in the kernel. Less performant thanxdp-nativebut more widely compatible.xdp-offloaded: The NIC hardware itself runs the eBPF program. Requires specific NIC hardware and driver support.
xdp.rxQueueDiscount: Forxdp-nativeandxdp-offloaded, this can reduce the number of Rx queues the kernel needs to poll, further optimizing performance.xdp.scheduler: For load balancing, this specifies the algorithm (e.g.,roundrobin,random,maglev).
Here’s how you might configure a service to use XDP for load balancing. Assuming xdp.mode=xdp-native is set cluster-wide:
apiVersion: v1
kind: Service
metadata:
name: my-xdp-service
annotations:
io.cilium/xdp-load-balancer: "true" # This annotation is often implicit with XDP enabled cluster-wide
spec:
selector:
app: my-app
ports:
- protocol: TCP
port: 80
targetPort: 8080
type: LoadBalancer
When io.cilium/xdp-load-balancer: "true" (or simply when XDP is enabled and the service is of type LoadBalancer or NodePort), Cilium will automatically program its XDP eBPF maps to handle the load balancing for this service. The lb4: map will be populated with the backend pod IPs, and the XDP program will intercept and redirect traffic directly.
The one thing many people overlook is that XDP acceleration in xdp-native mode requires explicit support from the network interface card (NIC) driver. If your NIC driver doesn’t expose an XDP hook or doesn’t support the specific eBPF bytecode Cilium generates, xdp-native will fail to load. In such cases, Cilium will often fall back to xdp-generic, which works but sacrifices a significant portion of the performance benefits. Always verify your NIC and driver compatibility for true xdp-native performance.
After successfully enabling XDP acceleration and seeing your packet throughput skyrocket, the next challenge you’ll likely encounter is managing XDP’s behavior for traffic that isn’t destined for Kubernetes services, such as host-to-host traffic or traffic to/from the Kubernetes node itself.