An etcd cluster’s primary role is to be the single source of truth for distributed system state, and its availability is paramount because it’s the linchpin for virtually all distributed coordination.
Let’s spin up a production-grade etcd cluster, prioritizing TLS for security and High Availability (HA) for resilience. We’ll aim for a 3-node cluster, the minimum for HA to tolerate one node failure.
The Core Components:
- etcd Server: The actual etcd process that stores data and participates in consensus.
- etcdctl: The command-line client to interact with the etcd cluster.
- TLS Certificates: Essential for encrypting communication between etcd nodes and between clients and etcd. We need a Certificate Authority (CA) and individual certificates for each etcd member and for clients.
Prerequisites:
You’ll need three machines (physical or virtual) that can communicate with each other over the network. Let’s assume their IPs are 192.168.1.101, 192.168.1.102, and 192.168.1.103. We’ll also need a CA to sign our certificates. For simplicity, we’ll generate self-signed certificates.
Step 1: Generate Certificates
This is the most involved part. We need a CA, server certificates for each etcd node, and client certificates.
First, create a directory for your certs:
mkdir etcd-certs
cd etcd-certs
Now, generate the CA key and certificate:
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -out ca.crt -subj "/CN=etcd-ca"
Next, generate a client certificate and key for etcdctl:
openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr -subj "/CN=etcd-client"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 365 -sha256
Now, for each etcd node, generate its server certificate and key. We need to include all possible IP addresses and DNS names the node might be accessed by in the Subject Alternative Name (SAN) field.
Node 1 (192.168.1.101):
openssl genrsa -out node1.key 2048
openssl req -new -key node1.key -out node1.csr -subj "/CN=etcd-node1"
# Create a SAN config file for node1
cat <<EOF > node1.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage=digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
subjectAltName=@alt_section
[alt_section]
IP.1=192.168.1.101
DNS.1=node1.example.com # If using DNS names
EOF
openssl x509 -req -in node1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out node1.crt -days 365 -sha256 -extfile node1.ext
Repeat this for node2 (192.168.1.102) and node3 (192.168.1.103), adjusting nodeX.ext accordingly.
Finally, bundle the CA, client cert/key, and each node’s cert/key into a distribution you can copy to the respective nodes.
Step 2: Install etcd
Download the latest etcd release binary for your OS from the official etcd GitHub releases page. Place the etcd binary in /usr/local/bin/ on each of your three nodes.
Step 3: Configure and Start etcd on Each Node
We’ll start etcd on the first node, then add the others.
On Node 1 (192.168.1.101):
Copy ca.crt, node1.crt, node1.key to /etc/etcd/ on this node.
Create the etcd data directory: mkdir -p /var/lib/etcd.
Here’s the etcd command for node 1. This command initializes a new cluster:
sudo etcd \
--name node1 \
--data-dir /var/lib/etcd \
--listen-client-urls https://192.168.1.101:2379,https://127.0.0.1:2379 \
--advertise-client-urls https://192.168.1.101:2379 \
--listen-peer-urls https://192.168.1.101:2380 \
--initial-advertise-peer-urls https://192.168.1.101:2380 \
--initial-cluster node1=https://192.168.1.101:2380,node2=https://192.168.1.102:2380,node3=https://192.168.1.103:2380 \
--initial-cluster-token etcd-cluster-1 \
--initial-cluster-state new \
--cert-file /etc/etcd/node1.crt \
--key-file /etc/etcd/node1.key \
--trusted-ca-file /etc/etcd/ca.crt \
--client-cert-auth true \
--peer-cert-file /etc/etcd/node1.crt \
--peer-key-file /etc/etcd/node1.key \
--peer-trusted-ca-file /etc/etcd/ca.crt \
--peer-client-cert-auth true
Explanation of Flags:
--name: Unique name for this etcd member.--data-dir: Directory to store etcd state.--listen-client-urls: URLs etcd listens on for client requests.https://enforces TLS.--advertise-client-urls: URLs clients should use to connect to this member.--listen-peer-urls: URLs etcd listens on for peer (inter-node) communication.--initial-advertise-peer-urls: URLs other members should use to connect to this member for peer communication.--initial-cluster: A comma-separated list of member names and their peer URLs for initial cluster bootstrap.--initial-cluster-token: A unique token to identify the cluster.--initial-cluster-state new: Indicates this is the first time bootstrapping this cluster.--cert-file,--key-file,--trusted-ca-file: TLS configuration for client-to-server communication.--client-cert-auth true: Requires clients to present a valid certificate signed by the CA.--peer-cert-file,--peer-key-file,--peer-trusted-ca-file: TLS configuration for server-to-server (peer) communication.--peer-client-cert-auth true: Requires peers to present valid certificates.
On Node 2 (192.168.1.102):
Copy ca.crt, node2.crt, node2.key to /etc/etcd/ on this node.
Create /var/lib/etcd.
sudo etcd \
--name node2 \
--data-dir /var/lib/etcd \
--listen-client-urls https://192.168.1.102:2379,https://127.0.0.1:2379 \
--advertise-client-urls https://192.168.1.102:2379 \
--listen-peer-urls https://192.168.1.102:2380 \
--initial-advertise-peer-urls https://192.168.1.102:2380 \
--initial-cluster node1=https://192.168.1.101:2380,node2=https://192.168.1.102:2380,node3=https://192.168.1.103:2380 \
--initial-cluster-token etcd-cluster-1 \
--initial-cluster-state existing \
--cert-file /etc/etcd/node2.crt \
--key-file /etc/etcd/node2.key \
--trusted-ca-file /etc/etcd/ca.crt \
--client-cert-auth true \
--peer-cert-file /etc/etcd/node2.crt \
--peer-key-file /etc/etcd/node2.key \
--peer-trusted-ca-file /etc/etcd/ca.crt \
--peer-client-cert-auth true
Notice --initial-cluster-state existing. This tells etcd to join the cluster defined by --initial-cluster.
On Node 3 (192.168.1.103):
Repeat the process for Node 3, copying ca.crt, node3.crt, node3.key and using --initial-cluster-state existing.
Step 4: Verify the Cluster
From any machine that has etcdctl and the client certificates (ca.crt, client.crt, client.key) copied over, run:
ETCDCTL_API=3 etcdctl \
--endpoints https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 \
--cacert=/path/to/etcd-certs/ca.crt \
--cert=/path/to/etcd-certs/client.crt \
--key=/path/to/etcd-certs/client.key \
endpoint health --cluster
You should see output indicating all members are healthy.
To see the cluster members:
ETCDCTL_API=3 etcdctl \
--endpoints https://192.168.1.101:2379 \
--cacert=/path/to/etcd-certs/ca.crt \
--cert=/path/to/etcd-certs/client.crt \
--key=/path/to/etcd-certs/client.key \
member list -w table
This will show you the IDs, status, name, peer URLs, and client URLs of all members.
Step 5: Running etcd as a Systemd Service
For production, you’ll want etcd to run as a systemd service. Create a service file (e.g., /etc/systemd/system/etcd.service) on each node. The service file will contain the etcd command with all its arguments. You’ll also need to manage the certificate distribution and ensure they are in the correct paths specified in the service file.
The initial-cluster-state flag is crucial. For the first node, it’s new. For all subsequent nodes joining an existing cluster, it must be existing. If you try to start a new node with new and the cluster already exists, etcd will refuse to start, preventing accidental cluster splits.
When adding a new member to an already running cluster (not part of the initial setup), you’d first start the new member with initial-cluster-state existing and point it to the existing members for peer communication. Then, you’d use etcdctl member add to inform the cluster about the new member, and it would be assigned a new ID and join the consensus.
The real magic of HA is that etcd uses the Raft consensus algorithm. This ensures that for any write operation, a majority of nodes must agree before the write is committed. If a single node fails, the remaining nodes (if still a majority) can continue to serve requests and maintain quorum. The initial-advertise-peer-urls is how nodes find each other to form this consensus group.