bpftool is your primary tool for peeking into the kernel’s eBPF runtime.
Let’s see it in action. Imagine you have a simple eBPF program attached to a network interface to count incoming packets.
# First, find the program's ID
bpftool prog list
# Output might look like this:
# 12345: kprobe name my_packet_counter tag abcdef0123456789
# 67890: xdp name drop_malicious_ips tag fedcba9876543210
# Let's say our packet counter is program ID 12345.
# Now, inspect its details:
bpftool prog show id 12345
# This will show you:
# ID: 12345
# Type: kprobe
# Name: my_packet_counter
# Tag: abcdef0123456789
# GPL license: true
# Loaded: ...
# Size: ... bytes
# Jited: ... bytes
# File: ...
# Function: ...
# Attach: ... (e.g., kprobe/eth0)
# Map IDs: 98765
The prog show command reveals crucial details: the program’s type (kprobe, xdp, tracepoint, etc.), its license (important for kernel compatibility), how much of it is JIT-compiled code, and importantly, the IDs of any eBPF maps it uses.
eBPF maps are where programs store and share data. They’re like in-kernel hash tables or arrays.
# Using the map ID from the previous output (98765):
bpftool map show id 98765
# Output might show:
# ID: 98765
# Type: hash
# Name: packet_counts_map
# Key type: u32 (size 4)
# Value type: u64 (size 8)
# Entries: 100
# Max entries: 4096
# Flags: 0
# ...
This tells us it’s a hash map, what data types its keys and values are, and its capacity.
To actually see the data inside a map, you’ll often lookup specific keys or dump its contents.
# To dump all entries from the map:
bpftool map dump id 98765
# Output might look like this:
# [{
# "key": [
# 0,
# 0,
# 0,
# 0
# ],
# "value": [
# 150,
# 0,
# 0,
# 0,
# 0,
# 0,
# 0,
# 0
# ]
# }, {
# "key": [
# 1,
# 0,
# 0,
# 0
# ],
# "value": [
# 75,
# 0,
# 0,
# 0,
# 0,
# 0,
# 0,
# 0
# ]
# }]
Here, key: [0,0,0,0] might represent CPU ID 0, and value: [150, ...] is the count of 150 packets. The value is shown as a byte array because eBPF maps can hold complex data structures, not just simple scalars.
You can also lookup specific keys. If you wanted to see the count for CPU 1 (which might be represented by the key [1,0,0,0]):
# This requires knowing the exact byte representation of your key.
# For a u32 key of 1, it's often 4 bytes, with the least significant byte first.
# So, 1 would be [1, 0, 0, 0].
bpftool map lookup id 98765 key '01 00 00 00'
# Output:
# value: [75 00 00 00 00 00 00 00]
The bpftool command is also your gateway to understanding program attachments.
# List all loaded eBPF programs and where they are attached
bpftool prog list -a
# Output might include:
# 12345: kprobe name my_packet_counter tag abcdef0123456789 <-- attached to kprobe/eth0
# 67890: xdp name drop_malicious_ips tag fedcba9876543210 <-- attached to xdp/eth0
The -a flag shows the attachment point. This is crucial for debugging why a program isn’t running or is running on the wrong events.
You can also inspect the relationships between programs and maps more directly.
# Show all maps and the programs that use them
bpftool map list --programs
# Output:
# 98765: hash packet_counts_map (value=u64)
# prog 12345 (kprobe my_packet_counter)
# Show all programs and the maps they use
bpftool prog list --maps
# Output:
# 12345: kprobe my_packet_counter
# map 98765 (hash packet_counts_map)
This cross-referencing is invaluable for understanding the dependencies in your eBPF deployment.
The most surprising truth about bpftool is its ability to help you understand the verifier’s perspective. While not directly showing verifier logs, you can use bpftool prog dump to see the low-level eBPF bytecode that was generated after compilation and verification. This bytecode is what the kernel actually executes.
# Dump the raw eBPF instructions for a program
bpftool prog dump id 12345
# This will output a long list of instructions like:
# 0: R1 = 0
# 1: R6 = 0
# 2: R2 = 0
# 3: R3 = 0
# 4: R4 = 0
# 5: R5 = 0
# 6: R7 = 16
# 7: R8 = 0
# 8: call <__ktime_get_ns+0>
# 9: R2 = R6
# 10: R3 = 0
# 11: R4 = 0
# 12: R5 = 0
# 13: R1 = *(u64 *)(R10 - 8)
# ... and so on.
By examining this bytecode, you can sometimes infer why a program might be behaving unexpectedly, or why the verifier might have rejected it (though bpftool itself doesn’t show verifier errors; you’d typically see those in dmesg). Understanding these instructions is key to mastering eBPF’s inner workings.
The next step in mastering eBPF observability is understanding how to trace eBPF programs themselves.