Geneve can encapsulate more metadata than VXLAN, making it more flexible for advanced networking features.
Let’s see Cilium’s VXLAN and Geneve tunnel modes in action.
Imagine you’re running a Kubernetes cluster and need to connect pods across different nodes. Cilium, a popular Kubernetes networking solution, offers two primary methods for encapsulating pod traffic to achieve this: VXLAN and Geneve. Both create overlay networks, essentially tunnels, that allow IP packets from pods to travel across the underlying physical network as if they were on the same L2 segment.
Here’s a simplified view of what happens when a pod on Node A wants to send traffic to a pod on Node B:
- Pod A sends a packet: The packet has the destination IP of Pod B.
- Node A’s network stack (managed by Cilium) intercepts it: It knows Pod B is not on Node A.
- Encapsulation: Cilium wraps the original pod packet inside another IP packet. This outer packet has:
- Source IP: Node A’s IP address.
- Destination IP: Node B’s IP address.
- A protocol number indicating it’s a tunnel (VXLAN or Geneve).
- Underlying Network: The physical network (switches, routers) forwards this outer packet based on Node A’s and Node B’s IPs.
- Node B receives the packet: It recognizes the tunnel protocol.
- Decapsulation: Node B’s network stack (Cilium) unwraps the outer packet, revealing the original packet from Pod A.
- Delivery: The original packet is then delivered to Pod B.
VXLAN (Virtual eXtensible LAN)
VXLAN is a mature and widely adopted tunneling protocol. It works by encapsulating Layer 2 Ethernet frames within UDP packets. The key identifier in VXLAN is the VXLAN Network Identifier (VNI), a 24-bit field that allows for up to 16 million unique virtual networks.
-
Configuration Example (Cilium
NetworkConfig):apiVersion: cilium.io/v2alpha1 kind: CiliumNetworkConfig metadata: name: example-config spec: tunnel: mode: vxlan vxlan: # You can optionally specify a specific VNI if needed, # otherwise Cilium will auto-assign. # vni: 4096
When Cilium is configured for VXLAN, it will use UDP port 4789 by default for its tunnels.
Geneve (Generic Network Virtualization Encapsulation)
Geneve is a more recent and flexible protocol designed to overcome some of VXLAN’s limitations. It also encapsulates Layer 2 frames within UDP but offers a richer set of options. Geneve uses a 24-bit Virtual Network Identifier (VNI) similar to VXLAN, but its primary advantage lies in its metadata option field. This field can carry additional information about the encapsulated packet, such as network type, tenant ID, or other custom metadata, making it highly extensible for future networking features or specific cloud provider requirements.
-
Configuration Example (Cilium
NetworkConfig):apiVersion: cilium.io/v2alpha1 kind: CiliumNetworkConfig metadata: name: example-config spec: tunnel: mode: geneve geneve: # Geneve also uses a VNI. # vni: 4096
Geneve typically uses UDP port 6081 (though this can be configured).
Which to Choose?
For most standard Kubernetes deployments, VXLAN is perfectly adequate and often the simpler choice. It’s well-supported, performant, and its VNI mechanism is sufficient for isolating tenant networks.
Geneve becomes the preferred choice when:
- You anticipate needing to carry richer metadata with your network traffic. This could be for advanced network policy enforcement, integrating with specific cloud provider network services, or enabling future Kubernetes networking features that require more context than VXLAN provides.
- Your underlying network infrastructure has specific requirements or preferences for Geneve.
- You are aiming for maximum future-proofing and flexibility in your cluster’s networking.
The primary difference you’ll observe in the tunnel packet itself is the protocol and the optional metadata. VXLAN is a simpler, more established protocol. Geneve, with its extensible metadata field, offers greater flexibility for advanced use cases and future network innovations.
The one thing most people don’t realize is that the VNI in both VXLAN and Geneve is not just an arbitrary number; it’s the key that the destination node uses to demultiplex incoming tunnel packets and correctly associate them with the target pod’s network namespace and Cilium’s policy enforcement. If VNIs aren’t unique across your cluster (or across different tunnel domains if you have multiple), you can get traffic misrouting or policy bypasses, as packets intended for one virtual network might be incorrectly processed by another.
Once you’ve chosen and configured your tunnel mode, the next challenge is often optimizing the MTU (Maximum Transmission Unit) for your overlay network to avoid packet fragmentation and performance degradation.