etcd members can be added or removed without downtime by leveraging its distributed consensus protocol to maintain quorum and data consistency throughout the operation.
Let’s see etcd in action. Imagine a cluster with three members: etcd-1 (10.0.0.1), etcd-2 (10.0.0.2), and etcd-3 (10.0.0.3).
# On etcd-1
ETCDCTL_API=3 etcdctl member list
# Output:
# 71e9c72e0245057f, started, etcd-1, https://10.0.0.1:2380, https://10.0.0.1:2379
# 8340e4968126d128, started, etcd-2, https://10.0.0.2:2380, https://10.0.0.2:2379
# a24f68c4a41f0695, started, etcd-3, https://10.0.0.3:2380, https://10.0.0.3:2379
# On etcd-2
ETCDCTL_API=3 etcdctl member list
# Output will be identical
The etcdctl member list command shows the current state of the cluster. Each member has a unique ID, a status (started, stopped, removed), a name, and URLs for peer-to-peer communication (for cluster membership) and client communication (for API requests).
Now, let’s add a new member, etcd-4 at 10.0.0.4.
First, start etcd-4 as a new member, pointing to the existing cluster. This requires knowing the peer URLs of at least one existing member. We’ll use etcd-1’s peer URL: https://10.0.0.1:2380.
# On the new etcd-4 machine
etcd \
--name etcd-4 \
--data-dir /var/lib/etcd \
--listen-peer-urls https://10.0.0.4:2380 \
--listen-client-urls https://10.0.0.4:2379 \
--advertise-client-urls https://10.0.0.4:2379 \
--initial-advertise-peer-urls https://10.0.0.4:2380 \
--initial-cluster etcd-1=https://10.0.0.1:2380,etcd-2=https://10.0.0.2:2380,etcd-3=https://10.0.0.3:2380,etcd-4=https://10.0.0.4:2380 \
--initial-cluster-state new \
--discovery '' # Important: if not using discovery service
When etcd-4 starts, it attempts to connect to the existing members. The existing members, in turn, will detect the new member and add it to their internal cluster membership list. This process happens asynchronously.
After etcd-4 has successfully started and joined, we can verify its inclusion:
# On etcd-1
ETCDCTL_API=3 etcdctl member list
# Output will now include:
# 71e9c72e0245057f, started, etcd-1, https://10.0.0.1:2380, https://10.0.0.1:2379
# 8340e4968126d128, started, etcd-2, https://10.0.0.2:2380, https://10.0.0.2:2379
# a24f68c4a41f0695, started, etcd-3, https://10.0.0.3:2380, https://10.0.0.3:2379
# b9a8c7d6e5f4a3b2, started, etcd-4, https://10.0.0.4:2380, https://10.0.0.4:2379
The key to adding members without downtime is that etcd’s Raft consensus algorithm can tolerate a certain number of nodes being unavailable. As long as a majority of members (quorum) are still operational and communicating, the cluster can continue to serve requests and make progress. Adding a new member doesn’t immediately disrupt the existing quorum; it’s a gradual process where the new member syncs up.
To remove a member, say etcd-3 (ID a24f68c4a41f0695), you first tell the cluster to remove it. This is done via the API.
# On etcd-1
ETCDCTL_API=3 etcdctl member remove a24f68c4a41f0695
# Output:
# Member a24f68c4a41f0695 removed from cluster <cluster-id>
This command instructs the remaining members of the cluster to mark etcd-3 as removed. They will stop trying to communicate with it for consensus. Crucially, the etcd process on etcd-3 itself is not stopped by this command. It will continue running but will eventually realize it’s no longer part of the cluster when it fails to establish consensus with the majority.
After issuing the member remove command, you should then stop the etcd process on the removed member (etcd-3) and clean up its data directory.
# On etcd-3 (if still running)
sudo systemctl stop etcd
# Then, optionally, remove data directory
sudo rm -rf /var/lib/etcd/
The cluster will continue to operate as long as quorum is maintained. Removing a member only affects the quorum if the cluster falls below the majority threshold. For a 3-member cluster, removing one member leaves 2, which is still a majority (2 > 3/2). For a 4-member cluster, removing one leaves 3, which is still a majority (3 > 4/2).
The most counterintuitive part of etcd cluster management is that the etcdctl member remove command only signals removal to the other members. It does not inherently stop the process or delete the data on the target node. This separation of concerns is vital: the cluster management layer tells the remaining nodes to ignore the departing member, while the node’s own process management and data cleanup are separate operational tasks. This design prevents accidental cluster-wide disruption if the remove command is issued incorrectly or if the target node is already down.
If you were to remove the last remaining member of a cluster, the cluster would effectively cease to exist, and any new member attempting to join would start a new, independent cluster.