FireLens makes it surprisingly easy to route your ECS logs anywhere, and the magic is all in the Fluent Bit configuration.
Let’s see it in action. Imagine you have a simple ECS service running a web application, and you want to send its logs to Amazon CloudWatch Logs.
Here’s a snippet of a task definition that sets up FireLens:
{
"family": "my-web-app-task",
"containerDefinitions": [
{
"name": "app",
"image": "nginx:latest",
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"config-file-type": "s3",
"config-file-value": "s3://my-firelens-configs/fluent-bit.conf"
}
}
},
{
"name": "fluent-bit",
"image": "amazon/aws-for-fluent-bit:latest",
"firelensConfiguration": {
"type": "fluentbit"
},
"essential": true
}
]
}
Notice the logConfiguration on the app container. We’re setting logDriver to awsfirelens. The options tell FireLens where to find our Fluent Bit configuration: in this case, a file named fluent-bit.conf stored in an S3 bucket. The fluent-bit container itself is declared as essential and its type is fluentbit, which is how FireLens knows to use it as the log router.
Now, let’s look at that fluent-bit.conf file. This is where the routing logic lives.
[SERVICE]
Flush 5
Daemon off
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/containers/*_app_*.log
Tag app.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[OUTPUT]
Name cloudwatch_logs
Match app.*
region us-east-1
log_group_name /ecs/my-web-app
log_stream_prefix ecs-app-
This configuration tells Fluent Bit:
[SERVICE]: Basic Fluent Bit settings.Flush 5means it will try to send logs every 5 seconds.[INPUT]: This section defines how Fluent Bit gets logs.Name tail: It will read log files from the filesystem.Path /var/log/containers/*_app_*.log: This is the crucial part for ECS. FireLens mounts the container logs into this directory on the Fluent Bit container’s filesystem. The pattern*_app_*.logtargets logs from containers namedapp.Tag app.*: Each log record processed by this input will be tagged withapp.*. This tag is used for routing.
[OUTPUT]: This section defines where to send the logs.Name cloudwatch_logs: We’re sending to CloudWatch Logs.Match app.*: This is the routing rule. If a log record has the tagapp.*(which our input does), it will be sent to this output.region us-east-1andlog_group_name /ecs/my-web-app: These specify the target CloudWatch Logs group.log_stream_prefix ecs-app-: This adds a prefix to the log stream name within the group, helping to organize logs.
When your ECS service starts, FireLens injects the Fluent Bit container and configures the app container to send its logs to Fluent Bit. Fluent Bit then reads these logs using the tail input, tags them, and routes them to CloudWatch Logs based on the Match rule.
The problem this solves is the inflexibility of the default ECS log drivers. With FireLens and Fluent Bit, you’re no longer limited to just sending logs to CloudWatch. You can configure Fluent Bit to send logs to S3, Splunk, Elasticsearch, Kafka, or virtually any destination that Fluent Bit supports, often with just a change to the [OUTPUT] section of your fluent-bit.conf.
Here’s the counterintuitive part: you might think the Path in the [INPUT] section is pointing to files inside your application container. It’s not. FireLens is a proxy; it intercepts the logs your application container would have sent to the default driver and instead forwards them to the Fluent Bit container. Fluent Bit then reads these logs from its own filesystem, typically from a path like /var/log/containers/, where FireLens makes them available.
The next thing you’ll want to explore is how to handle structured logs, like JSON, and how to perform transformations on your logs before sending them to their destination.