Elastic APM agents can leak sensitive data if not configured with proper authentication.
Let’s see it in action. Imagine you have a Node.js application. You’ve installed the @elastic/apm-agent package:
npm install @elastic/apm-agent
And you’ve configured it to send data to your Elastic APM server. The simplest, but insecure, way looks like this:
const apm = require('@elastic/apm-agent').start({
serviceName: 'my-node-app',
serverUrl: 'http://localhost:8200', // Your APM Server URL
});
This start() call, without any authentication, will send all your APM data – including potentially sensitive transaction names, URLs, and even request/response bodies if you’ve enabled full tracing – to your APM server. Anyone with access to that server can see it.
The problem Elastic APM is trying to solve is visibility into your application’s performance and errors. It traces requests, records database queries, captures exceptions, and aggregates this into a coherent view. But this data can be sensitive. For instance, a transaction name like /api/v1/users/{userId}/personal-details might expose internal data structures or PII if not sanitized.
Here’s how it works internally: The APM agent runs as a module within your application. It intercepts various events – HTTP requests, database calls, uncaught exceptions – and packages them into APM events. These events are then batched and sent over HTTP to the APM Server. The APM Server processes these events and stores them in Elasticsearch for analysis.
The key to securing this flow is to ensure that only authorized agents can send data to your APM Server. Elastic APM supports two primary mechanisms for this: Secret Tokens and API Keys.
Secret Tokens are simpler. They are a shared secret that the agent includes in its requests. The APM Server validates this token.
To use a secret token, you first generate one within Kibana. Navigate to Stack Management > Security > Tokens. Click Create token, give it a descriptive name (e.g., apm-agent-secret-token), set its permissions to allow apm_agent operations, and then click Create token. You’ll be shown the token – copy it immediately, as you won’t see it again.
Then, you configure your agent to use it:
const apm = require('@elastic/apm-agent').start({
serviceName: 'my-node-app',
serverUrl: 'http://localhost:8200',
secretToken: 'YOUR_GENERATED_SECRET_TOKEN', // Add this line
});
When the agent sends data, it will include an Authorization: SecretToken YOUR_GENERATED_SECRET_TOKEN header. The APM server checks this header. If the token is valid and has the correct permissions, the data is accepted. If not, the APM server will reject the request with a 401 Unauthorized or 403 Forbidden error.
API Keys offer more granular control. They are tied to specific users or service accounts and can have their permissions defined precisely. For APM, you’ll want an API key with the apm_agent role.
To create an API key for APM:
- Go to Stack Management > Security > API Keys.
- Click Create API key.
- In the "Name" field, enter something like
apm-node-app-key. - Under "Permissions", click Add permission.
- Select APM Agent from the dropdown.
- For "Actions", choose Write APM data.
- Click Create API key.
- Copy the generated Key ID and Key secret. Treat the secret like a password.
Configure your agent with the API key:
const apm = require('@elastic/apm-agent').start({
serviceName: 'my-node-app',
serverUrl: 'http://localhost:8200',
apiKey: 'YOUR_API_KEY_ID:YOUR_API_KEY_SECRET', // Add this line
});
The agent will send this as an Authorization: ApiKey YOUR_API_KEY_ID:YOUR_API_KEY_SECRET header. The APM Server validates both parts of the key and checks the associated permissions.
Using API keys is generally preferred for production environments because of their superior flexibility and auditability. You can revoke an API key at any time without affecting other agents or services using different keys.
The APM Server itself needs to be configured to accept these tokens/keys. If you’re using the default apm-server.yml configuration, it often has a secret_token or api_key section. For example, in apm-server.yml:
apm-server:
host: "localhost:8200"
secret_token: "YOUR_APM_SERVER_SHARED_SECRET" # If using secret tokens for server-to-server auth
api_key:
enabled: true
# If you want to restrict which API keys can be used, you can specify them here
# allowed_api_keys:
# - "YOUR_API_KEY_ID:YOUR_API_KEY_SECRET"
If your APM Server is behind a proxy or firewall, ensure that the serverUrl configured in your agent can reach the APM Server’s listening port (e.g., 8200).
When you use secretToken, the APM Server validates that the token presented by the agent matches one of the configured secret tokens. The APM Server doesn’t need a secret token itself for this validation, but rather the token that was generated and given to the agent. The secret_token in apm-server.yml is for authenticating the APM Server itself if it needs to communicate with Elasticsearch or Kibana using specific credentials. For agent authentication, you configure secret_token or api_key within the agent’s start() options.
The primary mechanism for securing the agent-to-APM Server communication is through the secretToken or apiKey configuration options within the agent. The APM Server, by default, accepts any valid token or API key that has been granted the apm_agent role or equivalent permissions. You can further restrict this on the APM Server side if needed, but the agent-side configuration is where you provide the credentials.
The next hurdle you’ll face is ensuring your APM data is properly sampled to avoid overwhelming your Elasticsearch cluster with high-volume applications.