Elastic APM’s major version upgrades are notorious for their potential to break things, but upgrading from 7.x to 8.x without losing any APM data is entirely achievable if you understand the key differences and follow a rigorous process. The most surprising thing is that the core data model for APM documents barely changed, but the way Elastic Stack components interact with that data—specifically, how indices are managed and queried—is fundamentally different, leading to compatibility issues if not handled with care.
Let’s see this in action. Imagine you have a typical APM setup:
- APM Server: Receives data from agents.
- Elasticsearch: Stores the APM data.
- Kibana: Visualizes the data and provides the APM UI.
- APM Agents: Instrument your applications.
The primary challenge in an upgrade like this isn’t usually the APM agents themselves (they have backward compatibility), but the seamless transition of data and the APM Server’s ability to continue writing to Elasticsearch and be read by Kibana.
Here’s a breakdown of what you need to do, focusing on the critical transition points:
1. Pre-Upgrade Planning and Backup
Before touching anything, back up your entire Elasticsearch cluster. This is non-negotiable. Use the Elasticsearch Snapshot API.
PUT _snapshot/my_backup_repository/snapshot_before_apm_upgrade
{
"indices": "apm-*",
"ignore_unavailable": true,
"include_global_state": false
}
This ensures you have a complete, point-in-time recovery if anything goes sideways.
2. Understanding the APM Data Model and Indexing Strategy
In Elastic APM 7.x, APM data was often stored in time-based indices like apm-7.16.0-traces-000001. In 8.x, the default indexing strategy shifts towards data streams. Data streams abstract away the underlying time-series indices, simplifying management and improving performance. This transition is where most compatibility breaks occur. The APM Server in 8.x expects to write to data streams, and Kibana’s APM UI expects to query them.
3. Upgrading the Elastic Stack Components
You must upgrade your Elastic Stack components in a specific order:
- Elasticsearch: Upgrade Elasticsearch first. This is the data store, and APM Server 8.x needs to communicate with Elasticsearch 8.x.
- APM Server: Upgrade the APM Server to version 8.x.
- Kibana: Upgrade Kibana to version 8.x.
Crucial Step: Migrating to Data Streams
The APM Server 8.x will not automatically migrate your old 7.x indices to 8.x data streams. You need to perform this migration explicitly.
a. Identify your 7.x APM indices:
GET _cat/indices/apm-*?v
You’ll see indices like apm-7.16.0-traces-000001, apm-7.16.0-errors-000001, etc.
b. Reindex your 7.x data into new 8.x data streams:
This is the most complex part and requires careful planning. You’ll need to reindex your historical data from your old 7.x indices into new data streams that the APM Server 8.x will manage.
First, create the data streams. The APM Server 8.x will create these automatically when it starts up and is configured correctly, but it’s good practice to have them ready or at least understand their naming convention. For example, traces-apm.traces-000001.
Then, use the _reindex API. You’ll do this for each data type (traces, errors, metrics, etc.).
Example for Traces:
Let’s say your old indices are apm-7.16.0-traces-* and you want to reindex into the traces-apm.traces data stream.
POST _reindex
{
"source": {
"index": "apm-7.16.0-traces-*"
},
"dest": {
"index": "traces-apm.traces",
"op_type": "create"
}
}
Explanation:
source.index: Specifies all your old 7.x trace indices.dest.index: Specifies the target data stream for your 8.x APM data. Theop_type: "create"is important here if you are writing to a data stream for the first time.
You will need to run this for apm-*errors-*, apm-*metrics-*, etc., mapping them to their respective data streams (error-apm.error, metrics-apm.metrics).
Important Considerations for Reindexing:
- Downtime: Reindexing can be resource-intensive and take a long time for large datasets. You’ll want to perform this during a maintenance window or plan for it to run in the background while your APM Server 7.x is still operational.
- Mapping: Ensure your data stream mappings are compatible. APM Server 8.x usually sets up correct mappings for its data streams. If you encounter mapping conflicts during reindex, you might need to adjust the target mapping or the source data.
- Version Compatibility: Always reindex from 7.x indices to 8.x data streams after Elasticsearch is upgraded to 8.x.
4. Configuring and Starting the APM Server 8.x
Once Elasticsearch is upgraded and you’ve begun reindexing, configure your APM Server 8.x. Key configuration points:
output.elasticsearch.hosts: Point to your new Elasticsearch 8.x cluster.output.elasticsearch.username/password: Use credentials that have appropriate permissions for Elasticsearch 8.x (including data stream management).apm.index_prefix: This should align with the new data stream naming convention (e.g.,apm). The APM Server will then use this to create and manage data streams liketraces-apm.traces,error-apm.error, etc.
Start the APM Server 8.x. It should begin writing to the new data streams.
5. Upgrading Kibana and Verifying APM Data
Upgrade Kibana to 8.x. Once it’s up and running:
- Check APM App: Navigate to the APM section in Kibana. It should now display data from the new data streams.
- Verify Data Continuity: Compare the data volume and time ranges in Kibana with what you expect from your historical data. You should see your reindexed data.
- Agent Configuration: Ensure your APM agents are configured to send data to the new APM Server 8.x. This is typically a simple URL change.
The most counterintuitive part of this entire process is realizing that the APM Server 8.x doesn’t migrate your old indices; it expects you to have already done that or to do it as part of the upgrade. It’s a manual reindexing task that bridges the gap between the old indexing strategy and the new data stream paradigm.
After successfully upgrading and reindexing, your next challenge will be optimizing query performance on the new data streams, which often involves understanding ILM (Index Lifecycle Management) policies applied to data streams.