Runtime threat detection in containers is surprisingly reactive, often waiting for an exploit to manifest rather than preventing it.
Let’s watch Falco, a popular open-source tool, in action. Imagine a container running a web server. We’ll use kubectl to deploy a simple Nginx pod:
apiVersion: v1
kind: Pod
metadata:
name: nginx-test
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
Now, let’s assume Falco is already running in your Kubernetes cluster, configured to monitor for suspicious activity. If someone inside this Nginx container were to try and execute a shell (/bin/sh) or, worse, try to write to sensitive system files like /etc/passwd, Falco would immediately detect it.
Here’s a simulated Falco alert you might see:
{
"output": "1:1234567890 - Critical: Shell spawned in container. Rule: 1000 - Shell was spawned in container. Container name: nginx-test. User: root (uid=0). Command: /bin/sh",
"priority": "Critical",
"rule": "Shell was spawned in container",
"time": 1678886400,
"output_fields": {
"container.id": "abcdef1234567890",
"container.name": "nginx-test",
"proc.cmdline": "/bin/sh",
"user.name": "root",
"user.uid": 0
}
}
This alert is triggered because the Falco rule "Shell was spawned in container" matches a common indicator of compromise. The output_fields give us the crucial context: the container name, the exact command executed, and the user ID.
Falco works by leveraging eBPF (extended Berkeley Packet Filter) programs, which are tiny, safe programs that run directly in the Linux kernel. Instead of relying on traditional kernel modules or user-space agents that often miss events or introduce performance overhead, eBPF allows Falco to tap into kernel system calls and network events with minimal performance impact. Think of it as a highly efficient, in-kernel observer.
When a system call occurs (like execve for spawning a process), the eBPF program attached to it can inspect the arguments. If these arguments match a predefined Falco rule (e.g., execve called with /bin/sh as the executable and the process is running inside a container), the eBPF program signals Falco. Falco then processes this signal, applies its rule engine, and generates an alert if a match is found.
The power of this approach lies in its depth. eBPF can observe not just process execution but also network connections, file access, and other kernel-level activities. Falco translates these low-level events into high-level security concerns. You can write rules to detect things like:
- A web server process attempting to open a reverse shell.
- A container trying to read sensitive files outside its designated volume.
- Unexpected network connections originating from a container.
- Modification of critical system binaries.
The configuration of Falco is done through YAML files, where you define rules. A basic rule might look like this:
- rule: Detect shell spawned in container
desc: A shell process was spawned in a container. This is often a sign of an attacker gaining access.
condition: >
container.runtime = 'docker' or container.runtime = 'cri'
and evt.type = 'open' and evt.dir = '<' and comm = 'sh'
output: Shell spawned in container. Container name: %(container.name)
priority: Critical
source: syscall
This rule triggers when a process named sh attempts to open a file (and the evt.dir check is a simplified representation; real rules are more nuanced) within a container environment (container.runtime is 'docker' or 'cri'). The source: syscall indicates that Falco is monitoring system call events.
What most people don’t realize is that eBPF’s programmability extends far beyond simple filtering. You can use eBPF to dynamically modify kernel behavior, redirect network traffic, or even implement entirely new security primitives at the kernel level, all without modifying the kernel source code itself. Falco, by using eBPF, is just one application of this powerful technology, focusing on security observability.
The next challenge you’ll face is correlating these runtime alerts with network security policies to understand the full scope of a potential breach.