Datadog’s Process Agent doesn’t just passively observe; it actively interrogates your running processes and their network interactions, painting a detailed picture of your system’s behavior.
Let’s see it in action. Imagine you have a simple web server running.
# On your web server host
# (Assuming you have Node.js installed)
npm init -y
echo "const http = require('http'); http.createServer((req, res) => { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello, Datadog!'); }).listen(8080, '127.0.0.1'); console.log('Server running on 127.0.0.1:8080');" > server.js
node server.js
Now, with the Datadog Agent and Process Agent installed and configured to collect process data (typically enabled by setting process_agent.enabled: true in datadog.yaml), you’d start seeing this Node.js process in your Datadog UI.
In the Datadog UI, navigate to Processes. You’ll see a table listing all detected processes, including the node process running your server.js. Clicking on this process row expands to reveal more details:
- Process Name:
node - Command Line:
node /path/to/your/server.js - User:
your_user - CPU Usage: (e.g.,
0.1%) - Memory Usage: (e.g.,
15MB) - Network Connections: This is where it gets interesting. You’ll see active connections. For our example, if you were to
curl http://127.0.0.1:8080from the same host, you’d see a connection from yourcurlprocess (orbash/zshif you’re not specifying the client directly) to127.0.0.1:8080on thenodeprocess. If your Node.js app were to make an outbound HTTP request to, say,api.example.com:443, you’d see that outbound connection here as well.
The Process Agent works by periodically querying the operating system’s process table and network connection information. On Linux, this typically involves reading from /proc filesystem entries (like /proc/[pid]/stat, /proc/[pid]/cmdline, /proc/[pid]/fd/ for network sockets). On Windows, it uses APIs like CreateToolhelp32Snapshot and Netstat equivalents. This raw data is then enriched with context (like correlating process IDs to executable names, command lines, and users) before being sent to the Datadog backend.
The core problem the Process Agent solves is providing visibility into what is running on your hosts and how it’s communicating, beyond just basic CPU/memory metrics. It answers questions like:
- "Which processes are consuming the most resources?"
- "Is my web server actually listening on the expected port?"
- "Which hosts are talking to each other on the network, and what processes are involved?"
- "Are there any unexpected processes running on my servers?"
You control the granularity of data collection through the datadog.yaml configuration file. Key settings include:
process_agent.enabled: true: Turns the agent on.process_agent.collect_connections: true: Gathers network connection data for each process.process_agent.max_per_minute: 100: Limits the number of process events sent per minute to avoid overwhelming the agent or network.process_agent.remote_config.enabled: true: Allows remote configuration updates from the Datadog UI.
You can also use tags to filter and group process data. For example, if you tag your hosts with env:production and service:webserver, process data originating from those hosts will automatically inherit those tags, allowing you to slice and dice your process inventory by environment or service.
The Process Agent leverages the standard OS process listing commands but goes much deeper by correlating network socket information directly to those processes. This means if you see a network connection on port 8080 in your system’s netstat output, the Process Agent can tell you exactly which executable and command line initiated or accepted that connection, and which user it was running as. It’s not just a list of connections; it’s a map of communication pathways tied to the specific applications generating them.
Once you’re comfortable with process and connection visibility, the next step is to start correlating this data with application-level metrics and traces using Datadog APM.