Read Cilium Policy Verdict Logs to Debug Dropped Packets (2026)

Cilium policy verdicts are the ultimate arbiter of whether a packet is allowed or denied, and understanding how to read these logs is key to debugging why your network traffic is getting dropped.

Let’s say you’re seeing packets dropped between pod-a in namespace-a and pod-b in namespace-b, and you suspect a Cilium policy is the culprit. Your first step is to find the Cilium agent logs on the node where pod-a is running.

The core of the problem lies in the fact that Cilium intercepts all network traffic and applies Network Policies (and other policy types) to decide what’s allowed. When a packet doesn’t match any allow rule and doesn’t fall under a deny rule either (which is less common for explicit drops), it can be implicitly denied if no policy explicitly permits it. The policy verdict logs tell you exactly which policy (or lack thereof) caused the drop.

Here’s how to dive into the logs:

First, identify the Cilium agent pod on the node where pod-a resides. You can usually find this by running kubectl get pods -n kube-system | grep cilium. Let’s assume it’s named cilium-abcdef.

Next, you’ll want to access the logs of this agent pod. The most effective way to get granular policy information is to increase the log level temporarily.

kubectl logs -n kube-system cilium-abcdef -f --level debug

Now, trigger the traffic from pod-a to pod-b again. You’ll be looking for log entries that mention "Policy Verdict" or "drop." The output will be verbose, so filtering is your friend. Try filtering for the IP address of pod-b or the destination port.

kubectl logs -n kube-system cilium-abcdef -f --level debug | grep "Policy Verdict"

Or, to be more specific to a potential drop:

kubectl logs -n kube-system cilium-abcdef -f --level debug | grep "drop"

When you see a relevant log line, it might look something like this:

... level=debug msg="Policy Verdict" src-pod="pod-a(namespace-a)" dst-pod="pod-b(namespace-b)" proto=TCP sport=54321 dport=80 verdict=DENY rule="<none>" reason="no matching policy"

This is the smoking gun. It tells you:

src-pod: The origin of the packet.
dst-pod: The intended destination.
proto: The protocol (e.g., TCP, UDP).
sport, dport: Source and destination ports.
verdict: Crucially, DENY.
reason: no matching policy. This is the key. It means no allow rule in any applicable Cilium Network Policy (or other policy like K8s NetworkPolicy, CiliumNetworkPolicy, etc.) explicitly permitted this traffic.

Common Causes and Fixes:

Implicit Deny (Most Common):
- Diagnosis: The log shows verdict=DENY and reason="no matching policy". This is the default behavior in Kubernetes networking and Cilium: if no policy explicitly allows traffic, it’s dropped.
- Fix: Apply a Cilium Network Policy (CNP) or Kubernetes NetworkPolicy that explicitly allows the desired traffic. For example, to allow pod-a to reach pod-b on TCP port 80:
```
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-pod-a-to-pod-b
  namespace: namespace-a # Or the namespace where pod-a resides
spec:
  endpointSelector:
    matchLabels:
      io.kubernetes.pod.name: pod-a
  ingress:
  - fromEndpoints:
    - matchLabels:
        io.kubernetes.pod.name: pod-b
        io.kubernetes.pod.namespace: namespace-b
    toPorts:
    - ports:
      - protocol: TCP
        port: "80"
```
  This policy, applied in namespace-a (or wherever pod-a is), allows pod-a to initiate connections to pod-b on TCP port 80.
- Why it works: Cilium evaluates policies based on selectors. This CNP selects pod-a and defines an ingress rule allowing traffic from specific endpoints (pod-b) to specific ports (80). Without this explicit allow, the traffic is implicitly denied by default.
Incorrect toPorts or protocol in Policy:
- Diagnosis: Logs show verdict=DENY but the reason might be more specific, or you see the correct policy matched but the port/protocol doesn’t align. The traffic might be going to TCP port 8080, but your policy only allows TCP port 80.
- Fix: Review your CNP or NetworkPolicy and ensure the toPorts and protocol fields accurately reflect the traffic being sent. For the example above, if pod-b listens on TCP 8080, change "port": "80" to "port": "8080".
- Why it works: Policies are exact matches. If the packet’s destination port or protocol doesn’t match any of the toPorts or protocol entries in an allow rule, that rule won’t permit the packet, leading to a potential deny.
Incorrect fromEndpoints or fromCIDR in Policy:
- Diagnosis: Similar to above, but the source is the issue. pod-a is trying to connect, but the policy on pod-b (if applied to the destination) or a cluster-wide policy doesn’t list pod-a (or its namespace/labels) in an ingress rule’s fromEndpoints or fromCIDR section.
- Fix: Adjust the fromEndpoints or fromCIDR in the relevant policy. If pod-a has the label app=frontend, and the policy on pod-b needs to allow it:
```
# Policy on namespace-b, selecting pod-b
spec:
  endpointSelector:
    matchLabels:
      io.kubernetes.pod.name: pod-b
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend # This label must match pod-a
        io.kubernetes.pod.namespace: namespace-a
    toPorts:
    - ports:
      - protocol: TCP
        port: "80"
```
- Why it works: Policies define who is allowed to initiate connections. If the source endpoint (identified by labels or CIDR) isn’t listed in an allow rule’s fromEndpoints or fromCIDR section, the traffic is not permitted by that rule.
Policy Applied to the Wrong Namespace or Endpoint:
- Diagnosis: You’ve written a policy, but it’s not being applied where you expect. For instance, a CNP with namespace: namespace-a might not be selecting pod-a if pod-a doesn’t have the labels specified in spec.endpointSelector. Or, you might be applying a policy to allow traffic to pod-b, but the policy is in the wrong namespace.
- Fix: Double-check the metadata.namespace and spec.endpointSelector in your CNP, and the metadata.namespace and spec.podSelector in your K8s NetworkPolicy. Ensure they correctly target the endpoint(s) you intend to control.
- Why it works: Cilium’s policy enforcement is namespace-aware and endpoint-selector driven. If the policy object isn’t in the same namespace as the endpoint it’s supposed to govern (unless using cluster-wide policies), or if the endpointSelector doesn’t match the labels on the pod, the policy simply won’t be applied to that pod.
Order of Operations with Multiple Policies:
- Diagnosis: You might have both Kubernetes NetworkPolicy and Cilium CiliumNetworkPolicy objects in play. Cilium respects K8s NetworkPolicies, but CNPs can offer more granular control. If a K8s NetworkPolicy denies traffic, a subsequent CNP allowing it might not be reached if the initial deny is final. Conversely, if a K8s NetworkPolicy allows traffic, but a CNP denies it, the CNP’s deny will take precedence.
- Fix: Understand the hierarchy. Kubernetes NetworkPolicies are evaluated first. If they deny, the packet is dropped. If they allow, Cilium’s CNPs are then evaluated. A DENY verdict from a CNP will drop the packet. Aim for clarity: use one type of policy for a given traffic flow, or ensure your combined policies don’t create conflicting rules.
- Why it works: Policy evaluation follows a specific order. For Kubernetes NetworkPolicies, if any NetworkPolicy selects the pod and has an ingress rule that doesn’t match, the packet is denied. If it does match, it’s allowed. Then, Cilium’s own policies are applied. A deny from Cilium will override an earlier allow from K8s.
Service Type and Policy Enforcement:
- Diagnosis: Sometimes, traffic might be dropped before it even reaches the pod’s IP if it’s being routed through a Kubernetes Service. Policies are applied at the pod IP level. If a Service is routing traffic to pod-b on a specific port, and the policy on pod-b is too restrictive, the packet will be dropped after leaving the Service but before entering pod-b.
- Fix: Ensure your policies are aligned with how Services are directing traffic. If a Service is sending traffic to pod-b on TCP port 80, the policy on pod-b must explicitly allow TCP port 80.
- Why it works: Services act as an intermediary. The packet arrives at pod-b’s IP address and port, but it’s the Cilium agent on that pod’s node that intercepts and applies policies. The Service’s existence doesn’t bypass Cilium’s policy enforcement for the destination pod.

After applying a fix, remember to reduce the log level back to info or default on the Cilium agent pod to avoid excessive log generation and potential performance impact:

kubectl logs -n kube-system cilium-abcdef --level info

The next error you’ll hit is likely a "connection refused" or "timeout" if the policy is now allowing traffic but the application on pod-b isn’t listening on the expected port, or if DNS resolution is failing.