Hubble Relay lets you see network traffic between pods, not just on a node.
Let’s see it in action. Imagine we have two nodes, node-1 and node-2, and we want to see traffic from pod-a on node-1 to pod-b on node-2.
First, ensure Hubble Relay is deployed and configured to collect flow data. This typically involves deploying the Hubble Relay as a DaemonSet or Deployment, ensuring it has the necessary network policies to receive flow data from Cilium agents, and configuring its output to a collector (like Elasticsearch, Kafka, or even a local file for debugging).
Here’s a snippet of a typical Hubble Relay deployment YAML (simplified):
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: hubble-relay
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: hubble-relay
template:
metadata:
labels:
k8s-app: hubble-relay
spec:
containers:
- name: hubble-relay
image: quay.io/cilium/hubble-relay:v0.12.0
args:
- --collector-addr=unix:///var/run/cilium/hubble.sock
- --listen-address=:4244
- --debug
volumeMounts:
- name: cilium-run
mountPath: /var/run/cilium
volumes:
- name: cilium-run
hostPath:
path: /var/run/cilium
The collector-addr points to the local Cilium agent’s Hubble socket, allowing Relay to capture flows originating from that node. The listen-address is where Relay itself will expose its API for querying.
To query this data, we use the hubble observe command. If Relay is running on node-1 and configured to expose its API on port 4244, we’d query it like this:
hubble observe --from-pod=<pod-a-namespace>/<pod-a-name> --to-pod=<pod-b-namespace>/<pod-b-name> --node=node-1 --port=4244
This command, when executed from a machine that can reach node-1’s port 4244, will show us the flow records for traffic initiated by pod-a and destined for pod-b, as seen by the Cilium agent on node-1.
The real power comes when you have multiple Relay instances, or a centralized Relay that aggregates data from multiple nodes. If you have a central Relay instance, say on node-0, listening on port 4244, and it’s configured to ingest data from node-1 and node-2, you can query it directly:
hubble observe --from-pod=<pod-a-namespace>/<pod-a-name> --to-pod=<pod-b-namespace>/<pod-b-name> --relay-ip=<node-0-ip> --relay-port=4244
This query would then show you the traffic as observed by the central Relay, which has potentially aggregated flows from various nodes.
The core problem Hubble Relay solves is providing visibility into inter-node pod communication that traditional network monitoring tools, often node-centric, miss. Cilium, by default, captures detailed L7 and L3/L4 flow data for pods it manages. Hubble Relay acts as a bridge, allowing you to access and query this rich data, even when the source and destination pods reside on different nodes. It leverages the eBPF capabilities within Cilium to capture these flows at the kernel level, providing granular visibility without requiring sidecar agents or complex network taps.
When configuring Relay, the collector-addr is crucial. It must point to the correct Cilium agent’s Hubble socket. On a standard Cilium installation, this is usually /var/run/cilium/hubble.sock mounted from the host’s /var/run/cilium directory. If your Cilium agent is running with a different socket path, Relay needs to be configured accordingly. The listen-address determines how you access Relay itself; for debugging, 0.0.0.0:4244 is common, but for production, binding to 127.0.0.1 or a specific IP is more secure.
The hubble observe command is your primary tool. You can filter by source and destination pods, namespaces, IPs, ports, and even L7 protocols. For instance, to see only HTTP traffic from pod-a to pod-b:
hubble observe --from-pod=<pod-a-ns>/<pod-a-name> --to-pod=<pod-b-ns>/<pod-b-name> --protocol=http --relay-ip=<relay-ip> --relay-port=4244
This command will then filter the captured flows to show only those matching the HTTP protocol, effectively giving you application-level visibility.
A common configuration pitfall is network policy. If your network policies are too restrictive, Cilium might not even capture the flow, or Relay might not be able to reach the Cilium agent’s socket. Ensure that the Relay pods (if deployed as a DaemonSet) have appropriate network policies allowing them to access the Cilium agent’s Hubble socket on each node. If using a centralized Relay, ensure the nodes can reach the Relay’s listen-address.
When Relay is configured to listen on 0.0.0.0:4244 and you’re trying to query it from your local machine, you might encounter issues if your Kubernetes nodes have strict egress firewall rules or if your local machine is not directly routable to the node’s IP. In such cases, port-forwarding is your friend:
kubectl port-forward -n kube-system ds/hubble-relay 4244:4244
This command forwards your local port 4244 to the hubble-relay DaemonSet’s port 4244. You can then use hubble observe --relay-ip=127.0.0.1 --relay-port=4244 to query the data.
The single most surprising thing about Hubble Relay’s data is that it’s not just about what traffic is flowing, but how it’s being processed by Cilium. You can see detailed information about the eBPF program execution for each flow, including any policy decisions made by Cilium, dropped packets, and even the specific L7 request/response details if TLS is not interfering. This level of introspection is unparalleled for Kubernetes networking.
Once you’ve mastered observing flows, the next step is often to start acting on them, perhaps by integrating Hubble’s flow data with Prometheus for metrics, or using it to drive automated responses to security events.