Falco rules aren’t just checklists; they’re actively shaping your security posture by deciding what constitutes an "event" and what’s just noise.
Let’s watch Falco in action. Imagine a common scenario: a web server process (nginx) suddenly starts shelling out to sh.
# Simulate a suspicious event
kubectl exec -it <your-pod-name> -- /bin/sh -c 'curl http://localhost:8080/exploit && /bin/sh -c "echo hello"'
Falco, with a well-tuned rule, would fire:
{
"output": "Shell spawned by unexpected process (containerd://<container-id>)",
"priority": "WARNING",
"rule": "Shell spawned by unexpected process",
"time": "2023-10-27T10:30:00.123456789Z",
"output_fields": {
"container.id": "<container-id>",
"container.name": "<pod-name>",
"container.runtime": "containerd",
"evt.caller.pid": 1234,
"evt.dir": "<",
"evt.pid": 5678,
"proc.cmdline": "/bin/sh -c echo hello",
"proc.exe": "/bin/sh",
"proc.name": "sh",
"proc.pid": 5678,
"rule.name": "Shell spawned by unexpected process",
"user.name": "root"
}
}
This alert tells us sh (the shell) was executed by containerd (the container runtime) within a pod named <pod-name>. The proc.cmdline shows the actual command that was run.
The core problem Falco rules solve is the signal-to-noise ratio. Without them, you’d be drowning in logs. With them, you get actionable alerts. Internally, Falco uses a powerful rule engine that analyzes system calls and other kernel events. Each rule is a condition that, when met, triggers an alert.
The key levers you control are:
condition: The logical expression that defines when an event is interesting. This is where you specify processes, users, file paths, network connections, etc.output: The human-readable message that appears in the alert. This can include variables from theoutput_fields.priority: The severity of the alert (e.g.,INFO,WARNING,ERROR).tags: Labels you can apply to rules for easier filtering and organization.
Let’s say you’re getting too many alerts for sh being spawned by kubectl exec. The default rule might be too broad.
# Example of a too-broad rule
- rule: Shell spawned by unexpected process
desc: A shell process was spawned by a suspicious parent process.
condition: proc.name = sh and proc.exe = /bin/sh and proc.cwd != /usr/local/bin and proc.cwd != /bin
output: Shell spawned by unexpected process (containerd://%{container.id})
priority: WARNING
tags: [shell, unexpected]
The problem here is proc.cwd != /usr/local/bin and proc.cwd != /bin is not specific enough to exclude kubectl exec. You want to allow sh when it’s spawned by kubectl but block it otherwise.
To tune this, you need to identify the actual parent process for kubectl exec within your container environment. Often, it’s containerd-shim or a similar process. You’d examine your Falco logs for alerts generated by kubectl exec and look at the parent.exe or proc.ppid fields.
Let’s refine the rule to exclude kubectl exec by checking the parent process name.
# Example of a more specific rule
- rule: Shell spawned by unexpected process (excluding kubectl exec)
desc: A shell process was spawned by a suspicious parent process, excluding kubectl exec.
condition: proc.name = sh and proc.exe = /bin/sh and proc.cwd != /usr/local/bin and proc.cwd != /bin and not proc.ppid in (select proc.pid from processes where proc.name = "containerd-shim")
output: Shell spawned by unexpected process (containerd://%{container.id})
priority: WARNING
tags: [shell, unexpected]
In this refined rule, we’ve added and not proc.ppid in (select proc.pid from processes where proc.name = "containerd-shim"). This tells Falco to only trigger the alert if the parent process ID (proc.ppid) of the spawned shell is not a containerd-shim process. This effectively silences the legitimate alerts from kubectl exec.
The select proc.pid from processes where proc.name = "containerd-shim" part is crucial. It dynamically queries the currently running processes to get the PIDs of containerd-shim. This is more robust than hardcoding PIDs, which change.
This technique of excluding known-good parent processes is fundamental. You’ll apply it to other rules where legitimate administrative actions trigger alerts. For instance, if a CI/CD pipeline’s build tools are spawning shells, you’d identify their parent processes and add them to the exclusion list.
The proc.cwd (current working directory) check is also a powerful filter. If you know that legitimate shell usage in your application containers never occurs in /tmp or /var/tmp, you can add and proc.cwd !~ "/tmp/.*" and proc.cwd !~ "/var/tmp/.*" to narrow down the scope. The !~ operator allows for regular expression matching, providing fine-grained control.
When tuning, always make small, incremental changes. Test each modification by intentionally triggering the behavior you want to allow or deny. Examine the Falco logs (/var/log/falco.log or your configured output) to see if your rule is behaving as expected.
The next challenge you’ll face is correlating these individual alerts into a cohesive incident, especially when multiple events occur in rapid succession.