APM events are just documents, and you can add custom fields to them with ingest pipelines just like you would any other document.
Let’s see it in action. Imagine you have a web service instrumented with the Elastic APM Ruby agent. By default, it sends requests, database queries, and external HTTP calls as APM events.
# config/initializers/elastic_apm.rb
ElasticAPM.configure do |config|
config.server_url = 'http://localhost:8200'
config.service_name = 'MyRailsApp'
config.secret_token = '' # If you have a secret token configured
config.environment = 'production'
end
Now, let’s say you want to add a custom field, user.id, to every transaction. This is useful for tracing requests originating from specific users.
First, you need to create an ingest pipeline in Kibana. Navigate to Stack Management > Ingest Pipelines. Click Create pipeline, give it a name (e.g., add_user_id_to_apm), and then add a Set processor.
In the Set processor configuration, under Fields, add:
-
Field:
user.id -
Value:
{{user_id}}
The {{user_id}} part is a Handlebars template. This means the pipeline will look for a field named user_id in the incoming APM event and use its value for user.id. If user_id doesn’t exist, user.id will be set to an empty string.
Next, you need to tell your APM integration to use this pipeline. Go to Observability > APM > Settings > Ingest pipelines. Find the pipeline you created (add_user_id_to_apm) and select it for Transactions, Spans, and Errors.
Now, how do you get that user_id field into the APM event in the first place? This is where your application code comes in. You need to add it to the APM event context. For the Ruby agent, you can do this within your controllers or request handling logic:
# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
before_action :set_user_id_for_apm
private
def set_user_id_for_apm
if current_user # Assuming you have a devise or similar authentication
ElasticAPM.set_label(:user_id, current_user.id)
end
end
end
The ElasticAPM.set_label(:user_id, current_user.id) call adds a label named user_id to the current APM transaction. This label is then picked up by the ingest pipeline.
When a transaction comes in, the APM Server first receives it. It then checks its configured ingest pipelines. If it finds the add_user_id_to_apm pipeline, it executes the Set processor. The processor looks for a field named user_id in the event data. Because your application code added this label, it’s available. The processor then sets the user.id field in the event to the value of user_id.
After the pipeline processes the event, the APM Server indexes it into Elasticsearch. You can then query for transactions, spans, or errors and filter or visualize based on the new user.id field. For example, in Kibana Discover, you could search for user.id : "123" to see all events associated with user ID 123.
The most surprising thing is that APM events are not special; they are just documents with a specific schema that the APM Server understands. You can apply any ingest pipeline processor to them, not just set. You could use gsub to sanitize a field, split to break a string into an array, or even enrich processors to add data from external lookups. The key is understanding that APM events flow through the same ingest pipeline mechanism as other Elasticsearch documents.
This flexibility allows you to enrich your APM data with business-specific context, making your monitoring far more powerful than just the default metrics. For instance, you could add a tenant.id field for multi-tenant applications or a feature_flag.name to track performance of specific feature flag variations.
The next challenge is handling dynamic values that aren’t directly available as labels, like deriving a custom field from an existing one using a regular expression.