Fluent Bit is the standard way to get your EKS pod logs into CloudWatch, but the setup can feel like a magic trick if you don’t know what’s going on under the hood.
Here’s a typical setup. You have a deployment for Fluent Bit in your EKS cluster, usually as a DaemonSet so it runs on every node. It watches for log files generated by other pods on that node.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: kube-system
spec:
selector:
matchLabels:
app: fluent-bit
template:
metadata:
labels:
app: fluent-bit
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.9.4 # Example image version
ports:
- containerPort: 2020
volumeMounts:
- name: varlog
mountPath: /var/log
- name: containers
mountPath: /var/lib/docker/containers # Or /var/log/containers for containerd
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
volumes:
- name: varlog
hostPath:
path: /var/log
- name: containers
hostPath:
path: /var/lib/docker/containers # Or /var/log/containers for containerd
- name: fluent-bit-config
configMap:
name: fluent-bit-config
The fluent-bit-config ConfigMap is where the magic really happens. It contains fluent-bit.conf, which defines how Fluent Bit processes logs.
[SERVICE]
Flush 5
Daemon On
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log # Adjust path based on your container runtime
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[OUTPUT]
Name cloudwatch_logs
Match kube.*
region us-east-1 # Your AWS region
log_group_name /aws/eks/my-cluster/pod-logs # Your CloudWatch log group name
log_stream_prefix ${NODE_NAME}- # Optional: prefix for log stream name
auto_create_group True
The [INPUT] section tells Fluent Bit to watch /var/log/containers/*.log (or wherever your container runtime stores logs) for new log entries. The Tag kub.* assigns a tag to these logs, which is then used in the [OUTPUT] section. The Parser docker tells Fluent Bit how to parse the JSON output from Docker/containerd, which includes Kubernetes metadata.
The [OUTPUT] section, specifically cloudwatch_logs, is configured to send logs tagged with kube.* to a specific CloudWatch Log Group. The region and log_group_name are crucial. auto_create_group True is a convenience that will create the log group if it doesn’t exist.
The fluent-bit container needs IAM permissions to write to CloudWatch Logs. This is typically done by attaching an IAM role to the EKS node instances that Fluent Bit pods run on. The role needs policies like CloudWatchLogsReadOnlyAccess and CloudWatchLogsWriteOnlyAccess (or more granular permissions targeting specific log groups).
The Parsers_File parsers.conf in the [SERVICE] section points to another ConfigMap that defines how to parse log lines. For container logs, you’ll often see a parser like this:
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
This parser tells Fluent Bit that the log lines are in JSON format and specifies the key and format for the timestamp. This allows Fluent Bit to correctly timestamp your logs in CloudWatch.
The most surprising thing about this setup is that Fluent Bit doesn’t actually read logs directly from the container’s stdout/stderr streams. Instead, it reads the log files that the container runtime (like Docker or containerd) writes to disk, typically in /var/lib/docker/containers/<container-id>/<container-id>-json.log or /var/log/containers/<pod-name>_<namespace>_<container-name>-<container-id>_<namespace>.log. Fluent Bit’s tail input plugin monitors these files for changes.
When you want to customize which logs go where, you’ll often modify the Match directive in the [OUTPUT] section. For example, Match app.my-app.* would only send logs tagged with app.my-app.* to that specific CloudWatch destination. You can also define multiple [OUTPUT] sections to send different logs to different CloudWatch Log Groups.
The log_stream_prefix is useful for organizing logs within a group. Using ${NODE_NAME}- will create separate log streams for each node, making it easier to trace logs back to their origin node.
The DB /var/log/flb_kube.db is Fluent Bit’s internal state file. It keeps track of which log lines have been successfully processed and sent. This prevents duplicate log entries if Fluent Bit restarts.
The Mem_Buf_Limit prevents Fluent Bit from consuming excessive memory if it’s unable to send logs to CloudWatch for an extended period.
The next step you’ll likely encounter is needing to filter logs before they are sent to CloudWatch, perhaps to reduce costs or noise.