DigitalOcean Insights doesn’t just show you Droplet metrics; it actively reconstructs them from low-level kernel events and network traffic.
Let’s see it in action. Imagine you’ve got a web server Droplet. You’re monitoring its CPU usage.
# Install the Insights agent
sudo apt-get update && sudo apt-get install digitalocean-insights
# Verify the agent is running
sudo systemctl status digitalocean-insights
Once installed, the agent starts collecting data. You can then view this in the DigitalOcean control panel under your Droplet’s "Insights" tab. You’ll see graphs for CPU, memory, disk I/O, and network traffic.
But what’s really happening under the hood? Insights hooks into the kernel’s eBPF (extended Berkeley Packet Filter) subsystem. This allows it to run tiny, safe programs directly within the kernel. When a process on your Droplet uses CPU, for instance, the kernel emits an event. Insights’ eBPF program intercepts this event, extracts the relevant process ID and CPU time consumed, and sends it to the Insights backend.
For network traffic, it’s similar. Insights uses eBPF to capture network packets at the kernel level, before they even hit userspace networking stacks. It analyzes packet headers to determine source, destination, port, and protocol, and aggregates this information. This means you see actual network throughput, not just what your application thinks it’s sending or receiving.
The problem Insights solves is providing accurate, low-overhead system monitoring without requiring intrusive agents that need to run as root in userspace, potentially impacting performance themselves. Traditional monitoring often relies on polling /proc or /sys files, which can be slow and less granular. eBPF allows for event-driven collection directly from the kernel’s source of truth.
Here’s the full mental model:
- eBPF Programs: Tiny, verified programs loaded into the kernel. They define what events to capture (e.g., process context switches, network packet arrivals) and what data to extract from those events.
- Kernel Hooks: eBPF programs attach to specific points in the kernel’s execution path (e.g.,
kprobes,tracepoints, network stack entry/exit points). - Data Collection: When an event occurs at a hooked point, the eBPF program executes, collects the specified data, and stores it in a shared memory map accessible by a userspace helper program.
- Userspace Agent: The
digitalocean-insightsdaemon on your Droplet reads data from these eBPF maps. - Aggregation & Transmission: The agent aggregates the collected raw data (e.g., summing CPU time per process over 10-second intervals) and securely transmits it to the DigitalOcean Insights backend.
- Visualization: The DigitalOcean control panel queries the backend and displays the aggregated metrics as graphs and tables.
You can control the granularity of data collection indirectly. While you can’t directly configure the eBPF programs, the Insights agent itself has settings for how frequently it polls the eBPF maps and how it aggregates data before sending it. These are generally tuned for optimal balance between detail and overhead. For instance, the agent might be configured to aggregate network packets into 1-minute buckets before uploading.
The Insights agent prioritizes collecting metrics that are truly indicative of system health and performance. It’s not just about raw numbers; it’s about understanding the behavior of your Droplet. For example, it can correlate network ingress with specific processes, allowing you to see which applications are consuming your bandwidth, something simple network monitoring tools might not easily expose without additional configuration. It also captures disk I/O at a per-process level, helping you pinpoint noisy neighbors on your storage.
The surprising part is how it reconstructs network traffic volume. It doesn’t just count packets. It uses eBPF to intercept packets at the network interface driver level. For each packet, it extracts its size from the packet buffer itself and adds that to a running total for the relevant flow, then aggregates these per second. This means you get an accurate byte count, even for protocols where packet size isn’t explicitly in the header.
Understanding how to correlate these metrics with application-level logs will be your next challenge.