Datadog’s cost model is primarily driven by ingested log volume, so filtering out noise before it hits Datadog is the most effective way to control your bill.
Let’s watch some logs flow through a common ingestion pipeline. We’ll use Fluentd as our log collector, outputting to a Datadog agent endpoint.
# fluentd.conf
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match **>
@type http
host api.datadoghq.com
port 443
scheme https
path /api/v1/logs
<http_header>
DD-API-KEY YOUR_DATADOG_API_KEY
</http_header>
<buffer>
flush_interval 5s
</buffer>
</match>
Now, imagine a busy application generating logs. If we don’t filter, everything goes to Datadog. This includes things like routine health checks, verbose debugging statements that aren’t needed long-term, or repetitive error messages that are already being alerted on.
The core problem we’re solving is cost. Every gigabyte of logs ingested has a price. By filtering, we reduce that volume. The secondary benefit is improved signal-to-noise ratio in Datadog, making it easier to find critical information.
Internally, Fluentd (or any log forwarder) reads logs from various sources, buffers them, and then sends them to the configured destination. Filters are plugins that operate on these logs between the source and the destination. They can inspect each log record and decide whether to keep it, modify it, or discard it entirely.
Consider a common scenario: application health checks. These might be requests to /health or /status endpoints that return a 200 OK and are generally uninteresting for long-term analysis. We can filter these out using Fluentd’s grep filter.
# fluentd.conf
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<filter **>
@type grep
<regexp>
key $.request.url
# Exclude logs where the URL is '/health' or '/status'
exclude true
/^\/health$|^\/status$/
</regexp>
</filter>
<match **>
@type http
host api.datadoghq.com
port 443
scheme https
path /api/v1/logs
<http_header>
DD-API-KEY YOUR_DATADOG_API_KEY
</http_header>
<buffer>
flush_interval 5s
</buffer>
</match>
This filter tells Fluentd: "For any log record, if the request.url field matches /^\/health$|^\/status$/, then exclude (drop) this record." This is highly effective for reducing traffic from load balancers or simple health check services.
Another common source of noise is verbose debug logging. While essential during development, it can overwhelm your Datadog account in production. You can filter based on log level.
# fluentd.conf
<filter **>
@type grep
# Keep logs that are NOT level 'debug'
<regexp>
key $.level
exclude true
/^debug$/
</regexp>
</filter>
This filter drops any log where the level field is exactly debug. You can expand this to include trace or other verbose levels. The key is to identify the field that reliably indicates the log’s verbosity.
Sometimes, you have specific fields that are highly repetitive and uninteresting. For instance, a user ID that cycles through a small set of test accounts. You can filter based on specific values.
# fluentd.conf
<filter **>
@type grep
# Exclude logs from internal test user 'testuser01'
<regexp>
key $.user_id
exclude true
/^testuser01$/
</regexp>
</filter>
This drops any log where the user_id field is testuser01. You can use regular expressions here to match patterns or multiple values.
A more advanced technique is to use the parser plugin to extract specific fields and then filter on those extracted fields. This is powerful when logs are unstructured. For example, if your application logs a JSON string within a general message field.
# fluentd.conf
<filter **>
@type parser
key message # The field containing the JSON string
<parse>
@type json
</parse>
</filter>
<filter **>
@type grep
# Now filter on a field that was INSIDE the JSON string
<regexp>
key $.internal_field.request_id
exclude true
/^abcde12345$/
</regexp>
</filter>
Here, the first filter parses a JSON string found in the message field, making its contents available as top-level fields. The subsequent grep filter can then operate on these newly extracted fields. This is crucial for breaking down complex log lines.
The rewrite_tag_filter plugin is another powerful tool. It allows you to change the tag of a log record based on its content, and then use different <match> blocks for different tags. This is effectively routing logs.
# fluentd.conf
<filter **>
@type rewrite_tag_filter
# If log contains 'important_event', change tag to 'important'
<rule>
key $.message
pattern /important_event/
tag important
</rule>
</filter>
# Default match for all other logs
<match **>
@type http
host api.datadoghq.com
port 443
scheme https
path /api/v1/logs
<http_header>
DD-API-KEY YOUR_DATADOG_API_KEY
</http_header>
<buffer>
flush_interval 5s
</buffer>
</match>
# Specific match for 'important' logs (e.g., higher retention, different processing)
<match important>
@type http
host api.datadoghq.com
port 443
scheme https
path /api/v1/logs
<http_header>
DD-API-KEY YOUR_DATADOG_API_KEY
</http_header>
# Maybe buffer differently, or send to a different Datadog intake endpoint if needed
<buffer>
flush_interval 1s
</buffer>
</match>
This allows you to send only critical logs to Datadog, or perhaps send less critical logs to a cheaper, long-term storage solution. The rewrite_tag_filter is the mechanism that enables this conditional routing.
The one thing most people don’t realize is how granular you can get with filtering. You’re not just filtering based on simple string matches; you can parse nested JSON, inspect individual fields within complex data structures, and even combine multiple conditions using regular expressions to create highly specific exclusion or inclusion rules. This level of control is what allows for significant cost savings without losing critical operational data.
After successfully filtering out all your noisy logs, the next problem you’ll encounter is ensuring your filters are comprehensive and not accidentally dropping important data.