Route Elastic APM Data to Elasticsearch or Kafka (2026)

Elastic APM data can be routed to Elasticsearch or Kafka, but the default behavior is to send it directly to Elasticsearch.

Here’s how to configure Elastic APM Server to send data to Kafka instead of Elasticsearch.

# apm-server.yml
output.kafka:
  hosts: ["kafka-broker-1:9092", "kafka-broker-2:9092"]
  topic: "apm-events"
  partitioner: "random" # or "round-robin"
  codec: "json" # or "deflate"

This configuration tells the APM Server to send all incoming APM data to the specified Kafka brokers on the apm-events topic. The partitioner setting determines how events are distributed across partitions, and codec specifies the data compression format.

Why would you do this?

The primary reason to route APM data to Kafka is to decouple the APM Server from Elasticsearch. This offers several advantages:

Buffering and Resilience: Kafka acts as a buffer. If Elasticsearch is temporarily unavailable or overloaded, APM data will be queued in Kafka, preventing data loss. The APM Server can continue accepting data, and the data will be processed by Elasticsearch once it’s back online.
Scalability: Kafka is designed for high-throughput, real-time data streams. By sending data to Kafka first, you can scale your Kafka cluster independently of your Elasticsearch cluster. This is especially useful if you have a very high volume of APM data.
Flexibility and Multiple Consumers: Kafka allows multiple consumers to read from the same topic. You could have your APM Server sending data to Elasticsearch, while a separate consumer application reads from the Kafka topic for custom analytics, anomaly detection, or integration with other systems without impacting the APM data flow to Elasticsearch.
Data Transformation and Enrichment: Before data lands in Elasticsearch, you can have a separate Kafka consumer application process and transform the APM data. This could involve adding additional metadata, filtering out unwanted fields, or enriching events with data from other sources.

How it works internally:

When you configure output.kafka, the APM Server switches its output mechanism. Instead of establishing connections to Elasticsearch and indexing documents, it connects to the Kafka brokers. Each APM event (transaction, span, error, etc.) is then serialized into the specified codec (e.g., JSON) and published as a message to the designated Kafka topic. The partitioner ensures that messages are distributed across the partitions of the topic, allowing for parallel processing by consumers.

Common configuration pitfalls:

Incorrect Kafka Broker Addresses: Ensure the hosts list contains valid and reachable Kafka broker addresses and ports. A typo here will prevent the APM Server from connecting.
Topic Doesn’t Exist: The specified Kafka topic must exist before the APM Server starts publishing to it. You might need to create the topic using Kafka’s command-line tools or your Kafka management interface. For example, using kafka-topics.sh:
```
kafka-topics.sh --create --bootstrap-server kafka-broker-1:9092 --replication-factor 3 --partitions 6 --topic apm-events
```
This command creates the apm-events topic with 6 partitions and a replication factor of 3.
Authentication/Authorization Issues: If your Kafka cluster requires authentication (e.g., SASL, SSL), you’ll need to configure these settings in apm-server.yml as well. This might involve specifying sasl.mechanisms, sasl.username, sasl.password, or SSL certificates.
Network Connectivity: Ensure that the APM Server has network access to the Kafka brokers. Firewalls or network segmentation can block these connections.
Data Serialization Mismatch: While json is common, if you choose a binary codec like deflate, ensure that your Kafka consumers are also configured to handle that codec.

Enabling Elasticsearch output alongside Kafka:

You can configure APM Server to send data to both Elasticsearch and Kafka simultaneously. This is useful for a phased migration or when you want both direct indexing and stream processing.

# apm-server.yml
output.elasticsearch:
  hosts: ["elasticsearch-node-1:9200"]

output.kafka:
  hosts: ["kafka-broker-1:9092"]
  topic: "apm-events"

In this scenario, APM Server will publish to both outputs. Elasticsearch will receive data for direct querying and dashboarding, while Kafka will receive it for stream processing.

The next step after routing to Kafka:

Once your APM data is flowing into Kafka, the next logical step is to set up consumers to process this data. This could involve deploying a dedicated consumer application that reads from the apm-events topic and indexes the data into Elasticsearch (if you’re still using it for analysis) or another data store, or it could be a custom application for real-time anomaly detection.