The etcd Maintenance API is your secret weapon for keeping your cluster healthy and recoverable, letting you reclaim space and create point-in-time backups without downtime.
Let’s see it in action. Imagine you have a healthy etcd cluster. You can check its health with a simple curl command:
curl -L http://127.0.0.1:2379/health
This will return {"health": "true"} if everything is peachy. Now, let’s say etcd has been running for a while, accumulating data. Over time, deleted keys leave behind "tombstones" and unreferenced data blocks, leading to a larger-than-necessary data directory. This is where defrag comes in.
The defrag operation reorganizes the data on disk, removing this unused space and potentially improving read/write performance. Crucially, it’s a non-disruptive operation. Your cluster keeps serving requests while defrag runs in the background. You trigger it via the Maintenance API:
curl -L -X POST http://127.0.0.1:2379/v3/maintenance/defrag
The API returns a json object indicating the status of the operation, like {"header":{"cluster_id":"...","member_id":"...","revision":"...","raft_term":"..."},"hash":0}. The hash value is a checksum of the database before defragmentation. You can run defrag again and compare the hashes to verify it worked.
Now, for disaster recovery, we have snapshot. A snapshot is a point-in-time backup of your entire etcd database. This is vital for restoring your cluster to a known good state if something goes catastrophically wrong. Like defrag, taking a snapshot is also non-disruptive.
To create a snapshot, you POST to the snapshot endpoint:
curl -L -X POST http://127.0.0.1:2379/v3/snapshot/save -o snapshot.db
This command saves the snapshot directly to a file named snapshot.db in your current directory. The -o flag redirects the output. The snapshot file is a binary representation of your etcd data. It’s crucial to store these snapshots securely and off the etcd nodes themselves.
The mental model here is that etcd is a distributed key-value store, and like any database, it needs maintenance. defrag is like database vacuuming, cleaning up internal fragmentation. snapshot is your tape backup, a reliable way to rewind time.
The etcd cluster is composed of Raft consensus groups. When you perform operations like defrag or snapshot, these are typically handled by the leader of the Raft group. The leader coordinates the operation across the cluster, ensuring consistency. For defrag, it’s about optimizing the underlying storage engine (MVCC - Multi-Version Concurrency Control). For snapshot, it’s about serializing the current state of that MVCC store into a consistent, readable format.
The etcd API endpoints are grouped under v3/maintenance for actions like defrag and compaction, and directly under /v3/snapshot for save and restore. You can also fetch the current status and configuration of your etcd cluster using the /v3/cluster/status and /v3/members/list endpoints respectively, which is good practice before performing any maintenance.
When you restore from a snapshot using etcdctl snapshot restore snapshot.db --data-dir /var/lib/etcd-restored, you are essentially initializing a new etcd data directory with the exact state captured in the snapshot file. This new directory can then be used to start a new etcd member, effectively joining your cluster (or starting a new one) from that historical point. It’s important to note that restoring a snapshot typically requires stopping the etcd process and replacing its data directory.
A common point of confusion is understanding the difference between defrag and compaction. While defrag cleans up physical disk space, compaction (triggered via etcdctl compact <revision>) purges historical revisions from the database’s MVCC store. You need to run compaction first to remove old revisions, and then defrag to reclaim the physical space left behind. Without compaction, defrag won’t have much to clean up in terms of historical data.
The next thing you’ll likely want to explore is how to automate these maintenance tasks and how etcd’s versioning and revision management tie into effective backup and recovery strategies.