Deploying Elastic APM on Kubernetes using a DaemonSet is a surprisingly straightforward way to get distributed tracing and application performance monitoring across your entire cluster.
Let’s see it in action. Imagine you have a simple Kubernetes deployment of a web application:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-webapp
spec:
replicas: 3
selector:
matchLabels:
app: my-webapp
template:
metadata:
labels:
app: my-webapp
spec:
containers:
- name: webapp-container
image: my-docker-repo/my-webapp:latest
ports:
- containerPort: 8080
To instrument this, we’ll deploy the Elastic APM Agent as a DaemonSet. This means one APM agent pod will run on each node in your Kubernetes cluster.
Here’s a sample DaemonSet manifest for the Elastic APM Agent:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: elastic-apm-agent
namespace: elastic-apm # Ensure this namespace exists
spec:
selector:
matchLabels:
app: elastic-apm-agent
template:
metadata:
labels:
app: elastic-apm-agent
spec:
containers:
- name: apm-agent
image: docker.elastic.co/apm/apm-agent-kubernetes:1.18.0 # Check for the latest version
env:
- name: ELASTIC_APM_SERVER_URLS
value: "http://elasticsearch-master.elastic-apm.svc.cluster.local:8200" # Your APM Server URL
- name: ELASTIC_APM_SECRET_TOKEN
valueFrom:
secretKeyRef:
name: apm-server-secret
key: secret-token
- name: ELASTIC_APM_SERVICE_NAME
value: "my-webapp" # This will be overridden by pod annotations
- name: ELASTIC_APM_ENVIRONMENT
value: "production" # This will be overridden by pod annotations
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 64Mi
# This is crucial for the agent to find the application's processes
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
# Mount the host's proc filesystem to allow the agent to inspect processes
volumes:
- name: proc
hostPath:
path: /proc
- name: sys
hostPath:
path: /sys
containers:
- name: apm-agent
# ... (image, env, resources as above)
volumeMounts:
- name: proc
mountPath: /host/proc
readOnly: true
- name: sys
mountPath: /host/sys
readOnly: true
The magic happens when you annotate your application pods. The APM agent running on the same node will automatically detect these annotations and start injecting itself into the application’s processes.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-webapp
spec:
replicas: 3
selector:
matchLabels:
app: my-webapp
template:
metadata:
labels:
app: my-webapp
annotations:
elastic.co/apm: "true" # This annotation tells the agent to hook into this pod
elastic.co/apm-service-name: "my-webapp" # Overrides the DaemonSet's default
elastic.co/apm-environment: "staging" # Overrides the DaemonSet's default
spec:
containers:
- name: webapp-container
image: my-docker-repo/my-webapp:latest
ports:
- containerPort: 8080
This DaemonSet approach bypasses the need to modify your application’s Dockerfile or deployment scripts to include the APM agent. The agent runs as a sidecar on the host, and uses capabilities like ptrace (enabled by the hostNetwork: true and /proc mount) to dynamically attach to and instrument your application’s processes. It effectively acts as a host-level agent that discovers and instruments eligible workloads based on annotations.
The primary problem this solves is achieving consistent, cluster-wide APM instrumentation without coupling your application deployments to the APM agent’s lifecycle or build process. The agent’s configuration (like the APM Server URL) is managed centrally in the DaemonSet, while workload-specific details (service name, environment) are applied via annotations directly on the application deployments. This separation of concerns makes managing APM across a dynamic Kubernetes environment much more scalable.
A key detail often overlooked is the hostNetwork: true setting on the APM agent DaemonSet. This allows the agent to directly access the host’s network stack, which is essential for it to communicate with the APM Server and, more importantly, to inspect and instrument processes running on the host’s network namespace. Without it, the agent would be confined to its own network and wouldn’t be able to effectively discover or interact with application processes.
When you deploy this setup, you’ll start seeing traces for your my-webapp service appear in the Elastic APM UI, showing requests, durations, and any errors occurring within your application instances across all nodes.
The next step after this is often configuring distributed tracing across multiple microservices, which involves ensuring each service has its own APM agent configured and that trace context is propagated between them.