Cilium monitor isn’t just a packet capture tool; it’s a live, event-driven stream of what Cilium’s data plane is actually doing with your network traffic, filtered through its policy and identity system.
Let’s see it in action. Imagine you have a simple application with two pods, frontend-1 and backend-2, and a policy that should allow frontend-1 to talk to backend-2 on port 8080.
Here’s how you’d start cilium monitor:
cilium monitor -t drop
This command tells Cilium to only show us packets that are being dropped by its data plane. Now, if frontend-1 tries to reach backend-2 on port 8080 and something goes wrong, cilium monitor will light up.
Let’s say frontend-1 (IP 10.0.0.1) tries to connect to backend-2 (IP 10.0.0.2) on port 8080. If that connection fails due to a policy violation, you might see something like this in your cilium monitor output:
Datapath drop: from frontend-1 (id=123) to backend-2 (id=456) on port 8080, reason: Policy denied (L4)
This is gold. It tells you exactly which endpoint (identified by its Cilium identity, not just IP) was the source, which was the destination, the port, and the reason for the drop: "Policy denied." This immediately points you to your Cilium Network Policies (CNPs).
But cilium monitor can show much more than just drops. You can watch different event types using the -t flag:
-t drop: As we saw, shows packets dropped by the datapath.-t policy: Shows packets that are evaluated against policies, including allowed and denied ones. This is useful to see if traffic is even reaching the policy enforcement point.-t nat: Displays Network Address Translation (NAT) events, crucial for understanding how Pod IPs are mapped to external IPs (or vice-versa) for egress and ingress.-t capture: This is the closest totcpdump, showing raw packets as they enter and leave the Cilium datapath.
Let’s build the mental model. Cilium operates at the eBPF level in the Linux kernel. When a packet arrives at a network interface managed by Cilium, it passes through a series of eBPF programs. These programs are responsible for:
- Identity Translation: Mapping IP addresses to Cilium’s internal identity system. This is fundamental because policies are enforced based on identities, not IPs.
- Policy Enforcement: Checking if the packet’s source identity is allowed to communicate with the destination identity on the specified port and protocol, according to your CNPs.
- NAT: Performing source NAT (SNAT) for egress traffic from pods and destination NAT (DNAT) for ingress traffic destined for services.
- Forwarding: Directing the packet to its next hop, whether that’s another pod on the same node, a pod on a different node, or out of the cluster.
cilium monitor taps into this pipeline. The -t flags select which stage’s events you want to see. The "reason" field in the output is particularly informative because it maps directly to the eBPF logic that made the decision.
Consider debugging ingress to a service. You have a frontend-1 pod outside the cluster trying to reach a backend-service running on backend-2. Your cilium monitor -t nat might show:
Datapath NAT: DNAT for service backend-service (ClusterIP 10.0.1.100:8080) to 10.0.0.2:8080
This confirms that Cilium is correctly performing the DNAT for the service. If this NAT event isn’t appearing, you’d look at your Kubernetes Service definition and your Cilium KubeProxyReplacement configuration (if applicable). If the NAT happens but the packet is still dropped, you’d then switch cilium monitor to -t drop to see the policy enforcement failure.
One of the most powerful, yet often overlooked, aspects of cilium monitor is its ability to show L3-L7 policy decisions in real-time. While -t policy shows L3/L4 events, if you’re using Cilium’s Hubble integration for L7 visibility (e.g., HTTP policies), cilium monitor can also surface these higher-level decisions. For instance, an HTTP policy might deny a request based on a specific URL path. You might see an event indicating a drop due to an L7 policy violation, providing context beyond just IP and port.
If you’re seeing unexpected connection resets or timeouts, and cilium monitor -t drop isn’t showing any explicit policy denials, the next thing to investigate is packet loss or misrouting within the Cilium datapath itself, which cilium monitor -t capture can help diagnose by comparing ingress and egress packet sequences.