Falco alerts can be sent to Slack and PagerDuty using a webhook.
Here’s a live example of a Falco alert and how it might be processed to send notifications.
Imagine Falco detects an unusual process execution:
{
"output": "10:52:54.175850394: Unexpected process called /bin/sh (user=root, command=sh -c echo hello > /tmp/test.txt, executable=/bin/sh, pid=12345, parent_pid=6789, tty=pts/0, group=root, user_uid=0, group_uid=0, proc_exe=/usr/bin/bash, proc_cmdline=bash -c echo hello > /tmp/test.txt)",
"priority": "warning",
"rule": "Unexpected shell spawned by bash",
"time": 1678882374175850394,
"ruleId": "1000000",
"source": "syscall",
"tags": ["process", "shell", "unexpected"],
"grpc_address": "127.0.0.1:5060",
"container": {
"id": "a1b2c3d4e5f67890abcdef1234567890abcdef1234567890abcdef1234567890",
"name": "my-app-container",
"image": "my-repo/my-app:latest",
"privileged": false,
"mounts": [
{"from": "/var/lib/docker/volumes/my-volume/_data", "to": "/app/data", "type": "volume"}
],
"labels": {
"io.kubernetes.pod.name": "my-app-pod-xyz",
"io.kubernetes.container.name": "my-app-container"
}
},
"fd": {
"type": "ipv4",
"name": "10.1.2.3:80"
},
"proc": {
"pid": 12345,
"ppid": 6789,
"exe": "/bin/sh",
"args": "sh -c echo hello > /tmp/test.txt",
"cwd": "/app",
"comm": "sh"
},
"user": {
"name": "root",
"uid": 0,
"gid": 0
}
}
This alert, triggered by the Unexpected shell spawned by bash rule, indicates that a shell process (/bin/sh) was executed by another process (/usr/bin/bash) within a container named my-app-container. The specific command executed was sh -c echo hello > /tmp/test.txt. This could be a sign of a compromised container or an unintended script execution.
To handle this, you’d configure Falco’s falco.yaml to output alerts to a webhook.
# falco.yaml
webhooks:
- enabled: true
url: "http://your-alert-manager:9090/webhook/falco"
# You can add authentication if needed:
# basic_auth:
# username: "user"
# password: "password"
The http://your-alert-manager:9090/webhook/falco is a placeholder for an actual alert processing service. This service will receive the JSON payload from Falco. A common choice for this is Alertmanager, which can then route alerts to Slack and PagerDuty based on your defined configurations.
Within Alertmanager, you’d have a receiver for Slack and another for PagerDuty.
Slack Receiver Configuration (alertmanager.yml):
receivers:
- name: 'slack-notifications'
slack_configs:
- api_url: '<YOUR_SLACK_WEBHOOK_URL>'
channel: '#falco-alerts'
send_resolved: true
title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} - {{ .CommonLabels.instance }}'
text: '{{ range .Alerts }}*Alert:* {{ .Annotations.summary }}<{{ .GeneratorURL }}|:link:>\n*Description:* {{ .Annotations.description }}\n*Falco Rule:* {{ .Labels.falco_rule }}\n*Container:* {{ .Labels.container_name }}\n*Pod:* {{ .Labels.pod_name }}\n{{ end }}'
Here, api_url is your Slack Incoming Webhook URL, and channel is the Slack channel where notifications will appear. The title and text fields define the message format, using Go templating to pull information from the alert.
PagerDuty Receiver Configuration (alertmanager.yml):
- name: 'pagerduty-notifications'
pagerduty_configs:
- routing_key: '<YOUR_PAGERDUTY_ROUTING_KEY>'
send_resolved: true
description: '{{ .CommonAnnotations.summary }}'
severity: '{{ .CommonLabels.severity }}'
details:
falco_rule: '{{ .CommonLabels.falco_rule }}'
container_name: '{{ .CommonLabels.container_name }}'
pod_name: '{{ .CommonLabels.pod_name }}'
command: '{{ .CommonAnnotations.command }}'
routing_key is your PagerDuty Events API v2 Integration Key. Alertmanager maps fields like description and severity to PagerDuty’s expected format.
The most counterintuitive aspect for many is how Alertmanager translates the raw Falco JSON into structured labels and annotations that can be used by templating. Falco itself doesn’t inherently know about "alertnames" or "summaries" in the way Alertmanager expects for routing. Instead, you configure Falco’s output format or use a tool like falco-exporter to transform Falco’s JSON into Prometheus-compatible metrics and alerts. Alertmanager then consumes these Prometheus alerts. The labels and annotations in the Alertmanager configuration are crucial for this mapping. For instance, you might configure Falco to include alertname and summary directly in its JSON output, or rely on a processor to add them.
The next step is to configure routing rules within Alertmanager to decide which alerts go to which receiver.