Elasticsearch’s Index Lifecycle Management (ILM) policies are the modern, built-in way to manage time-based indices, obsoleting the need for external tools like Curator.
Let’s see ILM in action. Imagine we have logs from our web servers, and we want to keep recent logs in hot, performant storage, move older logs to warm storage, and eventually delete logs older than 30 days.
Here’s a sample ILM policy:
PUT _ilm/policy/my_log_policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "1d",
"max_primary_shard_size": "50gb"
}
}
},
"warm": {
"min_age": "1d",
"actions": {
"allocate_replica_count": {
"number_of_replicas": 0
},
"forcemerge": {
"max_num_segments": 1
}
}
},
"cold": {
"min_age": "7d",
"actions": {
"freeze": {},
"allocate_replica_count": {
"number_of_replicas": 0
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}
This policy defines four phases: hot, warm, cold, and delete.
- Hot Phase: Indices start here. The
rolloveraction is key. It tells Elasticsearch to create a new index when the current one reaches either 1 day old (max_age) or 50 gigabytes in primary shard size (max_primary_shard_size). This ensures your active indices don’t grow indefinitely, impacting performance. - Warm Phase: Once an index is 1 day old, it moves into the warm phase. Here, we reduce the number of replicas to 0 to save resources, as this data is less frequently accessed.
forcemergeconsolidates segments, improving query performance on historical data. - Cold Phase: After 7 days, indices enter the cold phase. We
freezethem, which significantly reduces memory usage by unloading data from the JVM heap. Replicas are again set to 0. - Delete Phase: Finally, after 30 days, the
deleteaction removes the index entirely.
To make this policy active for new indices, you need to associate it with an index template. If you’re migrating from Curator, you likely have index patterns like logs-* or metrics-*. You’ll create or update a template for these patterns.
Here’s how you’d create a template:
PUT _index_template/my_log_template
{
"index_patterns": ["logs-*"],
"template": {
"settings": {
"index.lifecycle.name": "my_log_policy",
"index.lifecycle.rollover_alias": "logs-write"
}
}
}
The index.lifecycle.name setting links this template to our ILM policy. The index.lifecycle.rollover_alias is crucial for the rollover action. It specifies an alias that will always point to the current active index. When rollover occurs, Elasticsearch atomically switches the alias to the newly created index. Your applications should write to this alias, not directly to individual indices.
To set up the initial rollover index and alias, you’d run something like this:
PUT logs-000001
{
"aliases": {
"logs-write": {
"is_write_index": true
}
}
}
This creates the very first index (e.g., logs-000001) and assigns the logs-write alias to it, marking it as the one for writes. When the rollover conditions are met, a new index (e.g., logs-000002) will be created, and the logs-write alias will be updated to point to it.
The beauty of ILM is that it’s entirely managed within Elasticsearch. You don’t need to schedule cron jobs or external scripts. The state transitions are handled automatically by the cluster based on the policy’s min_age and action conditions. You can monitor the progress of your indices through the lifecycle in Kibana’s Stack Management > Index Lifecycle or by querying _ilm/explain for a specific index.
What most people don’t realize is that the forcemerge action in the warm phase can be quite resource-intensive. If you have many indices entering the warm phase simultaneously, it can impact cluster performance. It’s often better to schedule this during off-peak hours or to limit the number of concurrent forcemerge operations by adjusting the index.merge.scheduler.max_thread_count setting if you encounter issues.
The next step after mastering ILM is understanding how to manage data tiers (hot, warm, cold) across different hardware profiles using searchable snapshots and node attributes.