The most surprising thing about etcd backups is that you’re not just backing up data, you’re backing up the entire distributed state machine in a way that’s immediately replayable.
Let’s see etcd in action. Imagine a Kubernetes cluster where nodes are constantly joining and leaving, pods are being scheduled and terminated, and network policies are being updated. All of this state lives in etcd.
Here’s a simplified view of etcd’s internal operation. It uses the Raft consensus algorithm to ensure that all members of the cluster agree on the order of operations and the current state. When you write data to etcd, it’s first proposed to the leader, then replicated to a quorum of followers, and only then is it considered committed. Snapshots are essentially a point-in-time dump of this committed state, along with the Raft log up to that point.
The primary problem etcd backups solve is disaster recovery. If your etcd cluster becomes unavailable due to hardware failure, network partition, or human error, you need a way to restore it to a functional state. Snapshots provide this capability.
Here’s how you take a snapshot of an etcd cluster. We’ll use etcdctl, the command-line interface for etcd.
ETCDCTL_API=3 etcdctl snapshot save /var/lib/etcd/snapshot.db \
--endpoints=http://etcd-0.etcd.default.svc.cluster.local:2379,http://etcd-1.etcd.default.svc.cluster.local:2379,http://etcd-2.etcd.default.svc.cluster.local:2379 \
--cacert=/etc/etcd/pki/etcd-ca.crt \
--cert=/etc/etcd/pki/etcd.crt \
--key=/etc/etcd/pki/etcd.key
This command tells etcdctl to save a snapshot to /var/lib/etcd/snapshot.db. The --endpoints flag points to your etcd cluster members. The --cacert, --cert, and --key flags are crucial for authenticating with a TLS-enabled etcd cluster, which is standard practice for production environments.
The output of this command will be a single file, snapshot.db. This file contains all the keys and values stored in etcd at the moment the snapshot was taken, as well as the Raft log entries necessary to reconstruct the cluster’s state machine.
Restoring from a snapshot is a multi-step process. First, you need to stop all etcd members of the cluster you intend to restore to. Then, you can use the snapshot restore command.
ETCDCTL_API=3 etcdctl snapshot restore /var/lib/etcd/snapshot.db \
--data-dir=/var/lib/etcd/new-etcd-data \
--initial-cluster=etcd-0=http://etcd-0.etcd.default.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd.default.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd.default.svc.cluster.local:2380 \
--initial-cluster-token=my-etcd-cluster \
--initial-advertise-peer-urls=http://etcd-0.etcd.default.svc.cluster.local:2380 \
--name=etcd-0 \
--snapshot-restore-state=cluster
This command restores the snapshot into a new data directory, /var/lib/etcd/new-etcd-data. The --initial-cluster flag defines the members of the new cluster you are forming. --initial-cluster-token is a unique identifier for this cluster. --initial-advertise-peer-urls is how members will communicate with each other. Critically, --name specifies the name of the specific etcd member you are starting with this restored data. The --snapshot-restore-state=cluster flag is important; it tells etcd to restore the cluster metadata from the snapshot, not just the key-value data.
After restoring, you’ll need to start your etcd members with the restored data directory and appropriate configuration. The exact commands will depend on how you are running etcd (e.g., systemd, Kubernetes static pods). For a typical Kubernetes setup, you would update the volume mounts for your etcd pods to point to the new data directory.
One crucial aspect of etcd snapshots that often trips people up is understanding the snapshot-restore-state flag. If you omit it or set it to metadata, etcd will restore the key-value data but will treat the restored node as a new member joining an existing cluster, potentially leading to data divergence if the original cluster is still running. Using cluster ensures that the restored data forms the entire new cluster, effectively freezing the state at the time of the snapshot.
The next problem you’ll likely encounter is managing the lifecycle of these snapshots, such as automating their creation and rotation.