Generate Metrics from Logs in Datadog to Cut Indexing Costs (2026)

Datadog metrics derived from logs are not just a cost-saving hack; they fundamentally change the nature of your observability data, turning ephemeral events into persistent, queryable time-series.

Let’s see this in action. Imagine you have application logs like this, recording successful user logins and failed attempts:

{"timestamp": "2023-10-27T10:00:01Z", "level": "INFO", "message": "User login successful", "user_id": "alice123"}
{"timestamp": "2023-10-27T10:00:05Z", "level": "ERROR", "message": "Invalid password for user", "user_id": "bob456"}
{"timestamp": "2023-10-27T10:00:08Z", "level": "INFO", "message": "User login successful", "user_id": "charlie789"}
{"timestamp": "2023-10-27T10:00:12Z", "level": "ERROR", "message": "User not found", "user_id": "david007"}

Instead of indexing every single one of these log lines (which can get expensive), we can configure Datadog to extract specific pieces of information and aggregate them into metrics. This means we’re no longer paying to store every "User login successful" event; we’re storing a counter that increments whenever such an event occurs.

The core mechanism for this is Datadog’s Log Processing Pipelines, specifically the metrics processor. You define rules that parse your logs, and then, for specific parsed attributes, you tell Datadog to generate metrics.

Here’s how you’d set it up in Datadog:

1. Navigate to Log Processing: In the Datadog UI, go to Logs -> Configuration -> Pipelines.

2. Create or Edit a Pipeline: You’ll have a default pipeline, or you can create a new one. Pipelines are ordered, and the first one that matches a log will process it.

3. Add a Parsing Processor: If your logs aren’t already parsed (e.g., into JSON), you’ll need a JSON or Grokk processor first. For our example JSON logs, a JSON processor is perfect.

   • Processor Type: JSON
   • Source: content (assuming your log is in the content field)
   • Output Name: parsed_log (this is a temporary name for the parsed structure)

   After this, your log would look something like:

   {
     "timestamp": "2023-10-27T10:00:01Z",
     "level": "INFO",
     "message": "User login successful",
     "user_id": "alice123",
     "parsed_log": {
       "timestamp": "2023-10-27T10:00:01Z",
       "level": "INFO",
       "message": "User login successful",
       "user_id": "alice123"
     }
   }
   

4. Add a Metrics Processor: Now, we add the crucial step to generate metrics from the parsed data.

   • Processor Type: Metrics

   • Target: parsed_log.message (we want to base metrics on the log message content)

   • Metric Type: count (we want to count occurrences)

   • Metric Name: myapp.log.message.count (this will be the name of our new Datadog metric)

   • Tags: level:{{parsed_log.level}}, user_id:{{parsed_log.user_id}} (here’s where the real power is; we’re adding tags based on parsed log attributes, allowing us to slice and dice the metric)

   You can also specify filters. For example, to only generate metrics for successful logins, you’d add a filter to this processor:

   • Filter: parsed_log.message == "User login successful"

   Alternatively, you could create a separate metric for failures:

   • Processor Type: Metrics

   • Target: parsed_log.message

   • Metric Type: count

   • Metric Name: myapp.log.message.failure_count

   • Filter: parsed_log.level == "ERROR"

   • Tags: error_type:{{parsed_log.message}}

What happens under the hood?

When a log matching your pipeline arrives:

1. The JSON processor extracts the fields into the parsed_log structure.
2. The Metrics processor checks if the log’s parsed_log.message is "User login successful".
3. If it matches, it increments a Datadog metric named myapp.log.message.count. The current value of the level and user_id fields from the log are added as tags to this increment operation.
4. If the log’s parsed_log.level is "ERROR", it increments myapp.log.message.failure_count, tagging it with the specific error message.

Crucially, only the metrics are indexed and stored long-term, not the raw log lines that triggered them (unless you have other pipelines configured to index them). This dramatically reduces your indexing volume.

You can then visualize these metrics in Datadog dashboards just like any other metric. A query might look like:

• sum:myapp.log.message.count{level:INFO,service:my-app}
• sum:myapp.log.message.failure_count{service:my-app} by {error_type}

This allows you to track login success rates, error distributions, and other critical application behavior without the cost of indexing every single log event.

The one thing most people don’t realize is that the count metric type inherently creates a time-series. When you specify a metric name like myapp.log.message.count, Datadog doesn’t just store a single number; it creates a series of data points over time. Each time a log matching the rule arrives, it’s like calling increment() on a counter at that specific timestamp. This is why you can sum:, avg:, max:, min: these metrics over time ranges, and why they appear in your metrics explorer.

The next step is to explore how to generate distribution metrics from logs for latency or duration tracking.