Elastic APM Server can send its collected data to multiple destinations, but it’s not always obvious how to configure where that data goes or how it gets there.
Let’s see it in action. Imagine you’ve got APM data flowing into your server, and you want to send it not only to Elasticsearch for storage and visualization, but also to a Kafka topic for stream processing.
Here’s a snippet from apm-server.yml that accomplishes this:
output.elasticsearch:
hosts: ["elasticsearch:9200"]
index: "apm-%{+yyyy.MM.dd}"
output.kafka:
hosts: ["kafka:9092"]
topic: "apm-events"
codec:
json:
pretty: true
This configuration tells APM Server to send data to Elasticsearch at elasticsearch:9200, using an index pattern apm-YYYY.MM.DD. Simultaneously, it directs the same data to a Kafka broker at kafka:9092, publishing to the apm-events topic. The codec.json.pretty: true setting ensures the Kafka messages are human-readable JSON, which can be helpful during initial setup and debugging.
The core problem APM Server’s output configuration solves is decoupling data collection from data storage and processing. APM agents send data to APM Server. APM Server then outputs that data to one or more downstream systems. This allows you to:
- Centralize APM data: Agents from diverse services and environments all send to a single APM Server instance.
- Fan out data: Send the same APM data to multiple systems for different purposes (e.g., Elasticsearch for APM UI, Kafka for real-time analytics, S3 for long-term archiving).
- Manage data egress: Control how and where your sensitive performance data is stored and processed.
Internally, APM Server has a pipeline for processing incoming events. Once an event has been processed (e.g., enriched, validated), it’s handed off to the configured output plugins. Each output plugin is responsible for serializing the event into the format expected by the downstream system and then transmitting it.
The output section in apm-server.yml is your primary control panel. You can define multiple output.<type> blocks. Common types include:
elasticsearch: The default, sending data to Elasticsearch for use with Kibana’s APM UI.kafka: Sending data to Kafka topics.redis: Sending data to Redis lists.file: Writing data to local files (often used for debugging or local backup).log: Sending data to the APM Server’s own logs (again, for debugging).
Each output type has its own specific configuration options. For Elasticsearch, you’ll configure hosts, index, username, password, and SSL settings. For Kafka, you’ll specify hosts, topic, partition.roundrobin, codec, and authentication details.
When you configure multiple outputs, APM Server sends each incoming event to all configured output destinations. There’s no mechanism to split traffic or send different types of events to different outputs within the same APM Server instance. If you need that level of routing, you’d typically set up multiple APM Server instances, each with different output configurations.
The codec setting within an output configuration determines how the event is serialized. For elasticsearch, it’s typically JSON. For kafka, you can choose json or msgpack. The pretty: true option for JSON codec on Kafka is a handy shortcut for debugging, but it increases message size and CPU usage, so it should generally be disabled in production environments.
Transport settings, like SSL/TLS for connecting to Elasticsearch or Kafka, are configured within the respective output blocks. For example, to secure your Elasticsearch connection:
output.elasticsearch:
hosts: ["elasticsearch:9200"]
ssl.enabled: true
ssl.certificate_authorities: ["/etc/apm-server/certs/ca.crt"]
ssl.key: "/etc/apm-server/certs/client.key"
ssl.certificate: "/etc/apm-server/certs/client.crt"
This instructs APM Server to use TLS when connecting to Elasticsearch, specifying the certificate authority to trust, and providing client credentials if mutual TLS is required.
The power of APM Server’s output configuration lies in its flexibility. You can use it as a simple bridge to Elasticsearch, or as a central hub that fans out your critical observability data to a diverse set of backend systems for analysis, alerting, and long-term retention.
If you’re sending data to Elasticsearch and Kafka simultaneously, and you encounter issues where data appears in Elasticsearch but not Kafka, the first thing to check is the output.kafka section of your apm-server.yml for any typos in hosts or topic, and then verify that the Kafka broker is reachable from the APM Server.