Elastic APM’s architecture, when integrated with Fleet Server, allows for scalable, centralized management of APM data collection and forwarding.
Here’s how it looks in action. Imagine you have a fleet of microservices running across multiple Kubernetes clusters. Each service has the Elastic APM agent integrated.
# Example APM Agent Configuration (e.g., for Java)
elasticapm.service_name: my-awesome-service
elasticapm.server_urls: http://apm-server.elastic.co:8200
elasticapm.environment: production
Traditionally, each APM agent would report directly to an APM Server instance. With Fleet Server, this model shifts.
The APM agents are now configured to send their data to a Fleet Server instead of directly to the APM Server.
# Example APM Agent Configuration with Fleet Server
elasticapm.service_name: my-awesome-service
elasticapm.server_urls: http://fleet-server.elastic.co:8200
elasticapm.environment: production
Fleet Server acts as an intermediary. It receives data from all your APM agents and then forwards it to one or more backend Elastic Agent integrations, typically the APM Server integration.
This architecture solves several key problems:
- Scalability: Fleet Server can handle a massive volume of incoming APM data from numerous agents. It’s designed for high throughput and can be scaled horizontally by running multiple Fleet Server instances behind a load balancer. This prevents your APM Server from becoming a bottleneck.
- Centralized Management: Fleet Server, managed by Fleet, provides a single pane of glass for configuring and managing your APM agents and their data collection policies. You define "Integrations" in Fleet, which are pre-packaged configurations for APM agents, and Fleet pushes these configurations to the agents via the Fleet Server.
- Decoupling: It decouples the data ingestion point (Fleet Server) from the data processing and storage point (APM Server). This allows you to upgrade, scale, or even switch your APM Server instances without impacting the agents’ ability to send data.
- Security: Fleet Server can act as a secure gateway. Agents can connect to Fleet Server over TLS, and Fleet Server can then forward data to internal APM Servers, reducing the need to expose APM Servers directly to external networks.
Internally, Fleet Server uses Elastic Agent under the hood. When you set up the APM integration in Fleet, you’re essentially configuring an Elastic Agent policy. This policy dictates what data to collect, how to sample it, and where to send it. Fleet Server then acts as the "agent" on behalf of your APM agents, receiving their telemetry and re-packaging it for the APM Server integration.
The levers you control are primarily within Fleet:
- Agent Policies: You define policies that specify APM agent configurations (e.g.,
transaction_sample_rate,distributed_tracing_origins). - Integrations: You select and configure the "APM" integration within Fleet, pointing it to your APM Server backend.
- Data Streams: APM data lands in specific data streams (e.g.,
logs-apm.transaction,logs-apm.error,logs-apm.metric) within Elasticsearch, managed by the APM Server integration. - Fleet Server Deployment: You manage the scaling and resilience of your Fleet Server instances themselves.
One key aspect of this setup is how sampling and routing are managed. While agents might have initial sampling rules, Fleet Server can also apply routing logic or further filtering if configured. More importantly, the APM Server integration, once it receives data from Fleet Server, is responsible for parsing, enriching, and storing the data into its appropriate Elasticsearch data streams. This means the APM Server still performs critical post-processing, even though it’s receiving data indirectly. The configuration for how that data is processed and stored (e.g., index templates, ingest pipelines) is still managed by the APM Server integration’s configuration within Fleet.
The next step after understanding this architecture is to explore how to configure distributed tracing across multiple services using these components.