Kubernetes audit logs are the system’s eyewitnesses, but without a trained detective, they’re just noise.
Let’s see this in action. Imagine a user, alice, trying to deploy a malicious pod.
apiVersion: v1
kind: Pod
metadata:
name: evil-pod
spec:
containers:
- name: container
image: alpine
command: ["/bin/sh", "-c", "sleep 3600"]
securityContext:
privileged: true # This is a red flag!
When alice applies this:
kubectl apply -f evil-pod.yaml
The Kubernetes API server records this event. A typical audit log entry might look like this (simplified):
{
"level": "Metadata",
"auditID": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
"stage": "RequestReceived",
"requestInfo": {
"method": "POST",
"uri": "/api/v1/namespaces/default/pods",
"userAgent": "kubectl/v1.28.2 (linux/amd64) kubernetes/f1b35b2",
"remoteAddr": "192.168.1.100:54321"
},
"user": {
"username": "alice@example.com",
"uid": "alice-uid-123",
"groups": ["system:authenticated"]
},
"impersonatedUser": null,
"objectRef": null,
"requestBody": null,
"responseStatus": null,
"requestReceivedTimestamp": "2023-10-27T10:00:00Z",
"stageTimestamp": "2023-10-27T10:00:00.123Z"
}
And later, when the pod is actually created:
{
"level": "Request",
"auditID": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
"stage": "ResponseComplete",
"requestInfo": {
"method": "POST",
"uri": "/api/v1/namespaces/default/pods",
"userAgent": "kubectl/v1.28.2 (linux/amd64) kubernetes/f1b35b2",
"remoteAddr": "192.168.1.100:54321"
},
"user": {
"username": "alice@example.com",
"uid": "alice-uid-123",
"groups": ["system:authenticated"]
},
"impersonatedUser": null,
"objectRef": {
"kind": "Pod",
"namespace": "default",
"name": "evil-pod",
"apiVersion": "v1"
},
"requestBody": {
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "evil-pod",
"namespace": "default",
"creationTimestamp": null,
"labels": {}
},
"spec": {
"containers": [
{
"name": "container",
"image": "alpine",
"command": [
"/bin/sh",
"-c",
"sleep 3600"
],
"securityContext": {
"privileged": true
}
}
],
"volumes": null
},
"status": {}
},
"responseStatus": {
"code": 201
},
"requestReceivedTimestamp": "2023-10-27T10:00:00Z",
"stageTimestamp": "2023-10-27T10:00:01.500Z"
}
Falco, when configured to ingest these logs, can then analyze them. We’ll define a rule to detect the creation of privileged pods.
First, ensure your Kubernetes audit logs are being sent to a location Falco can access. This typically involves configuring the API server to write audit logs to a file or a webhook. For file-based logging, you’d modify the API server’s manifest (often found in /etc/kubernetes/manifests/kube-apiserver.yaml on control plane nodes) to include flags like:
spec:
containers:
- command:
- kube-apiserver
# ... other flags
- --audit-log-path=/var/log/kubernetes/audit.log
- --audit-policy-file=/etc/kubernetes/audit-policy.yaml
- --audit-log-maxsize=100 # MB
- --audit-log-maxbackup=10
# ...
And /etc/kubernetes/audit-policy.yaml would define what to log. A minimal policy to capture pod creation events might look like this:
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Request
resources:
- group: ""
resources: ["pods"]
verbs: ["create"]
Then, configure Falco to read this audit log. In falco.yaml, you’d add or modify the json_output and file_output sections, and ensure the rulesfile points to your custom rules. Crucially, you need to tell Falco to read the audit log file:
# falco.yaml
json_output: true
file_output:
enabled: true
keep_alive: true
syscall_event_fd: 0 # Use 0 to indicate stdin/pipe
filename: /tmp/falco_audit_events.log # Or wherever you want Falco's output
rulesfile:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
- /etc/falco/kubernetes_audit_rules.yaml # Your custom rule file
# Add this if you are reading from a file directly (less common for auditd)
# listen_address: file:///var/log/kubernetes/audit.log
The most common way to get audit logs into Falco is via stdout from the API server, piped to Falco. If your API server is configured to output audit logs to stdout (often via kubectl logs or a direct file if you’re managing it manually), you can pipe it:
kubectl logs -n kube-system <kube-apiserver-pod-name> -f --container kube-apiserver --previous | falco --audit-log-file -
Or if you’re running Falco as a DaemonSet, ensure its configuration points to the correct audit log location.
Now, let’s craft a Falco rule in kubernetes_audit_rules.yaml:
- rule: Kubernetes Privileged Pod Creation
desc: "Detects the creation of a privileged pod in Kubernetes. Privileged pods can bypass security restrictions and access host resources."
condition: >
k8s.audit.request.verb = "create" and
k8s.audit.request.resource.type = "pods" and
k8s.audit.request.object.spec.securityContext.privileged = true
output: >
Kubernetes privileged pod creation detected (user: %(k8s.audit.user.username)s,
namespace: %(k8s.audit.request.object.metadata.namespace)s, pod: %(k8s.audit.request.object.metadata.name)s).
priority: critical
source: k8s_audit
When alice creates the evil-pod, Falco will fire this alert:
10:01:05.500 [Alert] Kubernetes privileged pod creation detected (user: alice@example.com, namespace: default, pod: evil-pod). rule: Kubernetes Privileged Pod Creation priority: critical time:2023-10-27T10:01:05.500Z k8s.audit.request.verb: create k8s.audit.request.resource.type: pods k8s.audit.request.object.spec.securityContext.privileged: true k8s.audit.user.username: alice@example.com k8s.audit.request.object.metadata.namespace: default k8s.audit.request.object.metadata.name: evil-pod
This rule leverages Falco’s Kubernetes audit event schema. The k8s.audit.request.verb and k8s.audit.request.resource.type fields directly map to the API operation being performed. The k8s.audit.request.object.spec.securityContext.privileged field drills down into the pod’s specification to check for the privileged: true setting.
The magic here is Falco’s ability to parse and understand the structured JSON output from Kubernetes audit logs, treating each log entry as a distinct event that can be evaluated against predefined rules. This allows for real-time threat detection at the API level, independent of what’s happening inside the containers.
A common pitfall is not configuring the audit policy to capture enough detail. If your audit-policy.yaml is too restrictive (e.g., only logging Metadata level for pods), you might not get the requestBody containing the securityContext, and your rule will never fire. Ensure you’re logging at least Request level for the resources you care about.
The next thing you’ll likely want to tackle is detecting unauthorized access attempts or suspicious network activity originating from within your cluster.