etcd’s membership is static by default, meaning you have to manually add new members and then tell existing members about them. This is a critical step for scaling your Kubernetes cluster safely.
Here’s how to add a new etcd member to an existing cluster. Imagine we have a three-member etcd cluster already running, and we want to add a fourth member on a new machine.
First, you need to prepare the new etcd member’s machine. This involves installing the etcd binary and creating a configuration file. A minimal configuration file for the new member might look like this:
name: etcd-4
data-dir: /var/lib/etcd
listen-client-urls: http://192.168.1.4:2379
advertise-client-urls: http://192.168.1.4:2379
listen-peer-urls: http://192.168.1.4:2380
initial-advertise-peer-urls: http://192.168.1.4:2380
initial-cluster: etcd-1=http://192.168.1.1:2380,etcd-2=http://192.168.1.2:2380,etcd-3=http://192.168.1.3:2380
initial-cluster-state: existing
Let’s break this down.
name: A unique identifier for this new etcd node. We’re calling itetcd-4.data-dir: Where etcd will store its data.listen-client-urls: The addresses etcd will listen on for client requests (like from your Kubernetes API server).advertise-client-urls: The addresses other clients should use to reach this etcd node.listen-peer-urls: The addresses etcd will listen on for communication with other etcd members.initial-advertise-peer-urls: The addresses other etcd members should use to reach this node for peer communication.initial-cluster: This is the crucial part. It lists all members of the cluster, including the existing ones and the new one we’re about to add, along with their peer URLs. This tells the new member about the existing cluster.initial-cluster-state: existing: This tells etcd that it’s joining an existing cluster, not starting a new one.
Now, start the new etcd member with this configuration. For example, using systemd:
sudo systemctl start etcd
At this point, the new member etcd-4 knows about the existing cluster, but the existing members (etcd-1, etcd-2, etcd-3) don’t know about etcd-4. You need to inform them.
You do this by using the etcdctl command-line tool, which must be run from a machine that can reach the existing etcd cluster (often one of the existing etcd nodes or a control plane node).
First, set the ETCDCTL_API=3 environment variable to ensure you’re using the v3 API. Then, you’ll use the member add command. You need to provide the peer URL of the new member (http://192.168.1.4:2380) and its name (etcd-4). You also need to point etcdctl to the existing cluster’s client endpoints.
export ETCDCTL_API=3
etcdctl --endpoints=http://192.168.1.1:2379,http://192.168.1.2:2379,http://192.168.1.3:2379 member add etcd-4 --peer-urls=http://192.168.1.4:2380
This command will output something like:
Member 8175257073882049698 added to cluster abcdef1234567890
The output includes the ID of the new member (8175257073882049698) and the cluster ID (abcdef1234567890). This is important information.
The etcdctl member add command automatically updates the initial_cluster and initial_cluster_state fields in the new member’s etcd configuration file on disk. You don’t need to manually edit the config file on etcd-4 after running this command; etcdctl handles it. The new member will then attempt to connect to the existing cluster members using the peer URLs it now knows about.
After the new member has successfully joined, you need to update the existing members’ configurations to include the new member. This is a manual step for each existing node. You’ll edit their etcd configuration files (e.g., /etc/etcd/etcd.conf.yml or the systemd unit file).
For each existing member (e.g., etcd-1), you’ll append etcd-4=http://192.168.1.4:2380 to its initial_cluster setting.
So, if etcd-1’s initial_cluster was previously etcd-1=http://192.168.1.1:2380,etcd-2=http://192.168.1.2:2380,etcd-3=http://192.168.1.3:2380, you’ll change it to:
initial_cluster: etcd-1=http://192.168.1.1:2380,etcd-2=http://192.168.1.2:2380,etcd-3=http://192.168.1.3:2380,etcd-4=http://192.168.1.4:2380
After updating the configuration on each existing member, you must restart them.
sudo systemctl restart etcd
Once all existing members have been restarted with the updated configuration, they will recognize etcd-4 as a valid member of the cluster. You can verify this by running:
export ETCDCTL_API=3
etcdctl --endpoints=http://192.168.1.1:2379,http://192.168.1.2:2379,http://192.168.1.3:2379,http://192.168.1.4:2379 member list
You should see all four members listed with their IDs, status, and peer URLs.
The reason for this multi-step process is etcd’s raft consensus. When a new member joins, it needs to discover the existing cluster. The initial_cluster and initial_cluster_state=existing parameters on the new member allow it to bootstrap by reaching out to the known peers. However, for the cluster to be truly stable and resilient, all existing members must acknowledge the new member’s presence and include it in their own peer lists. This ensures that if any of the original members fail, the cluster can still elect a leader and maintain quorum with the new member participating.
The next challenge you’ll face is removing a member from the cluster, which requires a similar but inverse process.