An Elasticsearch snapshot is a point-in-time backup of your cluster’s indices and cluster state, allowing for disaster recovery and data migration.
Let’s see this in action. Imagine you have an Elasticsearch cluster running and want to back up its data to Amazon S3.
First, you need to set up an S3 bucket. Let’s say you create a bucket named my-elasticsearch-backups in the us-east-1 region.
Next, you configure Elasticsearch to trust Amazon’s certificate authority. This is usually handled by default on most Elasticsearch installations, but if you’re running in a highly restricted environment, you might need to add it manually.
Now, you need to grant Elasticsearch the necessary permissions to write to your S3 bucket. This is done by creating an IAM user with a policy that allows s3:PutObject, s3:GetObject, s3:ListBucket, and s3:DeleteObject actions on your specific bucket. You’ll then generate access keys (an Access Key ID and a Secret Access Key) for this user.
In Elasticsearch, you define a snapshot repository. This tells Elasticsearch where to store the backups. You’ll create a repository.json file like this:
{
"type": "s3",
"settings": {
"bucket": "my-elasticsearch-backups",
"region": "us-east-1",
"access_key": "YOUR_ACCESS_KEY_ID",
"secret_key": "YOUR_SECRET_ACCESS_KEY"
}
}
Then, you register this repository with your Elasticsearch cluster using the PUT _snapshot/my_s3_repository API call, with the repository.json file as the request body.
Once registered, you can take a snapshot. For example, to snapshot all indices, you’d use:
curl -X PUT "localhost:9200/_snapshot/my_s3_repository/snapshot_$(date +%Y%m%d%H%M%S)?wait_for_completion=true"
The wait_for_completion=true flag makes the request synchronous, meaning it won’t return until the snapshot is finished. The $(date +%Y%m%d%H%M%S) part dynamically generates a timestamp for the snapshot name, ensuring uniqueness.
Restoring is just as straightforward. To restore a snapshot named snapshot_20231027103000 to a new index called restored_my_index, you’d use:
curl -X POST "localhost:9200/_snapshot/my_s3_repository/snapshot_20231027103000/_restore" -H 'Content-Type: application/json' -d'
{
"indices": "my_original_index",
"rename_pattern": "my_original_index",
"rename_replacement": "restored_my_index"
}
'
This effectively copies the data from S3 back into your Elasticsearch cluster.
The core problem this solves is data durability and availability. Without snapshots, losing your Elasticsearch cluster means losing your data permanently. By storing snapshots in S3, you decouple your backups from your primary cluster’s infrastructure, providing an off-site, highly available, and cost-effective backup solution. S3’s durability guarantees ensure your backups are safe even if your entire AWS region experiences an outage.
The settings.readonly parameter in the repository configuration is often overlooked. Setting this to true prevents accidental deletion or modification of snapshots directly from Elasticsearch, enforcing that the S3 bucket remains the source of truth for backups and that any changes must be managed via S3’s own lifecycle policies or manual operations. This adds a crucial layer of protection against data loss due to misconfigurations or unauthorized access to your Elasticsearch cluster.
The next logical step is to automate snapshot creation and retention policies using S3 lifecycle rules.