Flame graphs are an incredibly powerful way to visualize CPU usage, but most people don’t realize they’re not just showing what is using CPU, but how that CPU time is being spent by the kernel on behalf of those processes.
Let’s see this in action. Imagine we have a simple Go program that’s busy doing some work:
package main
import (
"fmt"
"math/rand"
"time"
)
func busyWork() {
for i := 0; i < 1000000; i++ {
_ = rand.Intn(100)
}
}
func main() {
fmt.Println("Starting CPU intensive tasks...")
for {
go busyWork()
time.Sleep(10 * time.Millisecond) // Yield to let goroutines run
}
}
To profile this with eBPF and generate a flame graph, we’ll use bcc (BPF Compiler Collection). First, ensure you have bcc installed. Then, we’ll use the profile.py script that comes with bcc.
sudo /usr/share/bcc/tools/profile.py -p $(pgrep -f my_go_app) 100 10
Here’s what’s happening:
sudo: eBPF requires root privileges./usr/share/bcc/tools/profile.py: The script we’re using.-p $(pgrep -f my_go_app): This targets our Go application.pgrep -f my_go_appfinds the process ID (PID) of the Go program by searching its full command line. Replacemy_go_appwith the actual name or part of the command line of your application.100: This is the sampling frequency in Hertz (Hz). We’re telling the profiler to take a CPU sample every 100 times per second.10: This is the duration of the profiling in seconds. We’ll collect data for 10 seconds.
The script will output a text-based representation of the flame graph. You can then pipe this output to a tool like flamegraph.pl (also often found with bcc or available separately) to generate an interactive SVG:
sudo /usr/share/bcc/tools/profile.py -p $(pgrep -f my_go_app) 100 10 > profile.txt
/path/to/flamegraph.pl --flamechart < profile.txt > flamegraph.svg
Open flamegraph.svg in your browser. You’ll see a visualization where the width of each "flame" (or rectangle) represents the proportion of CPU time spent in that function. The wider a function is, the more CPU time it consumed. The top of the graph shows the functions at the top of the call stack (e.g., the entry point of the program), and functions called by them appear below.
The true power of eBPF here is that profile.py isn’t just looking at user-space function calls. It’s leveraging kernel probes. By default, profile.py samples the program counter (PC) of the CPU. When a sample is taken, it looks at the current PC and walks up the kernel’s stack trace for that CPU. This means you see not only your application’s user-space functions but also kernel functions that were executing on behalf of your process at the time of the sample. This includes things like system calls, scheduler functions, and even interrupt handlers if they were active.
For our Go program, you’d likely see busyWork and main as wide flames, but you’d also see functions related to the Go runtime scheduler (like runtime.goroutinego or runtime.schedule) and potentially kernel functions like do_nanosleep if the Go scheduler yields the CPU implicitly. The rand.Intn function might be a smaller flame within busyWork.
This ability to see both user-space and kernel-space execution in a single, unified view is what makes eBPF profiling so effective for diagnosing performance issues that span application and OS boundaries. You can spot if your application is bottlenecked by its own code, by inefficient system calls, or by contention within the kernel.
A common misconception is that flame graphs only show user-space code. This is true for older profiling tools like perf when configured to only sample user-space. However, eBPF tools like profile.py can, and often do, include kernel stacks by default. This means that a large, wide flame might not be your application code, but rather a kernel function like sys_futex if your application is frequently waiting on futexes, or __schedule if the scheduler is spending a lot of time deciding which process to run. You can distinguish these by looking at the function names – kernel functions typically start with sys_ or have other kernel-specific prefixes.
Once you’ve identified a performance bottleneck using flame graphs, the next step is often to understand the specific system calls or kernel interactions that are contributing to it, which might lead you to investigate tools like bpftrace for more targeted kernel event tracing.