Kubernetes pod network traffic tracing isn’t just about seeing packets; it’s about understanding the invisible dance of distributed systems.

Let’s watch this happen. Imagine a simple Nginx pod and a curl pod trying to talk to each other.

# nginx-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-demo
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
# curl-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: curl-demo
spec:
  containers:
  - name: curl
    image: curlimages/curl:latest
    command: ["sleep", "infinity"]

First, deploy these:

kubectl apply -f nginx-pod.yaml
kubectl apply -f curl-pod.yaml

Now, get their IPs:

NGINX_POD_IP=$(kubectl get pod nginx-demo -o jsonpath='{.status.podIP}')
CURL_POD_IP=$(kubectl get pod curl-demo -o jsonpath='{.status.podIP}')

echo "Nginx IP: $NGINX_POD_IP"
echo "Curl IP: $CURL_POD_IP"

From the curl-demo pod, let’s curl the Nginx pod:

kubectl exec curl-demo -- curl http://$NGINX_POD_IP

You’ll see the default Nginx welcome page. But what’s really happening?

This is where eBPF shines. It allows us to run sandboxed programs directly within the Linux kernel, without modifying kernel source code or loading kernel modules. For network tracing, this means we can tap into network events at incredibly low levels – before packets even hit userspace applications, and without the overhead of traditional packet capture tools like tcpdump or Wireshark.

The core idea is to attach eBPF programs to specific kernel hooks. For network traffic, these hooks are often found in the network stack: ingress/egress points of network interfaces, socket operations, or even the network datapath itself. When a packet or network event occurs at these hooks, your eBPF program gets a chance to inspect, modify, or simply record information about it.

Consider a common scenario: a request from curl-demo to nginx-demo.

  1. Socket Creation/Connection: When curl initiates a connection, a connect() syscall happens. An eBPF program attached to sys_enter_connect can see the destination IP and port.
  2. Packet Transmission: As the SYN packet leaves the curl-demo pod, it traverses the Kubernetes CNI network. An eBPF program attached to the egress path of the curl-demo pod’s network interface (e.g., xdp or tc ingress on the host’s veth pair) can see the packet’s source and destination MAC/IP addresses and payload.
  3. Packet Reception: The packet arrives at the nginx-demo pod’s network interface. An eBPF program on the ingress path of that interface sees the same packet.
  4. Application Handling: The packet reaches the Nginx process. An eBPF program could potentially trace socket receive/send buffers or even syscalls like read/write within the Nginx container.

Tools like cilium-cli or kubectl-trace (which leverages bcc or cilium/ebpf) abstract away much of the complexity. For instance, cilium-cli can deploy an eBPF-based agent that observes network events.

Let’s look at a specific eBPF program’s logic conceptually. A program might:

  • Hook: kprobe/tcp_connect or tracepoint/syscalls/sys_enter_connect.
  • Action: Read the sockaddr struct passed to the connect syscall. Extract the destination IP address and port.
  • Output: Send this information to userspace for logging.

Here’s a simplified bpftrace example (a high-level tracing language for eBPF) to see outgoing connections from pods:

# This requires bpftrace to be installed on your nodes
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_connect /comm == "curl" / { printf("curl connecting to %s:%d\n", args->uservaddr->sa_data, args->uservaddr->sa_family); }'

(Note: The uservaddr parsing can be complex and depends on the specific syscall and address family. For IP addresses, you’d typically parse sockaddr_in or sockaddr_in6 structures.)

When you run the kubectl exec curl-demo -- curl ... command again, you’d see output showing curl attempting to connect.

The one thing most people don’t realize is that eBPF programs can also filter packets before they even reach the application or the network stack, dramatically reducing the load on your system and simplifying the data you need to analyze. You can write programs that, for example, only allow packets with a specific destination IP and port to proceed, or drop packets that don’t match a certain pattern, all within the kernel.

Once you’ve mastered tracing individual pod connections, the next logical step is understanding how network policies are enforced and debugging inter-service communication failures in a multi-service environment.

Want structured learning?

Take the full Ebpf course →