etcd peers don’t actually verify each other’s identity by default when talking TLS, which is a bit like letting people into your house without checking their IDs, even though you locked the door.
Let’s see etcd in action with peer-to-peer TLS authentication.
Imagine you have a three-node etcd cluster: etcd-1, etcd-2, and etcd-3.
First, we need to generate certificates and keys for each node. This involves creating a Certificate Authority (CA) and then issuing certificates for each etcd peer signed by that CA.
# On a CA machine or a dedicated generation step
# Generate CA private key and certificate
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt -subj "/CN=etcd-ca"
# On each etcd node (e.g., etcd-1)
# Generate client key and certificate signing request (CSR)
openssl genrsa -out etcd-1.key 2048
openssl req -new -key etcd-1.key -out etcd-1.csr -subj "/CN=etcd-1.example.com"
# Sign the peer certificate with the CA
openssl x509 -req -in etcd-1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out etcd-1.crt -days 365 -sha256
# Repeat for etcd-2 and etcd-3, changing the CN in the CSR accordingly
# CN=etcd-2.example.com for etcd-2
# CN=etcd-3.example.com for etcd-3
Now, distribute ca.crt, etcd-1.crt, and etcd-1.key to etcd-1. Do the same for etcd-2 and etcd-3 with their respective certificates and keys.
The core of peer-to-peer TLS authentication lies in the peer-cert-file and peer-key-file flags, alongside trusted-ca-file and cert-allowed-cn.
Here’s how you’d configure etcd-1:
etcd \
--name etcd-1 \
--listen-peer-urls https://10.0.0.1:2380 \
--listen-client-urls https://10.0.0.1:2379 \
--advertise-client-urls https://10.0.0.1:2379 \
--initial-advertise-peer-urls https://10.0.0.1:2380 \
--initial-cluster etcd-1=https://10.0.0.1:2380,etcd-2=https://10.0.0.2:2380,etcd-3=https://10.0.0.3:2380 \
--initial-cluster-state new \
--peer-cert-file=/etc/etcd/pki/etcd-1.crt \
--peer-key-file=/etc/etcd/pki/etcd-1.key \
--trusted-ca-file=/etc/etcd/pki/ca.crt \
--cert-allowed-cn=etcd-1.example.com,etcd-2.example.com,etcd-3.example.com
Notice the crucial additions:
--peer-cert-file: The certificate for this specific etcd peer.--peer-key-file: The private key for this specific etcd peer.--trusted-ca-file: The CA certificate that signed all peer certificates. This is how etcd verifies the authenticity of the CA itself.--cert-allowed-cn: This is the key to peer authentication. It’s a comma-separated list of Common Names (CNs) that etcd will accept for its peers. In this example,etcd-1will only accept connections from peers whose certificates have a CN ofetcd-1.example.com,etcd-2.example.com, oretcd-3.example.com.
When etcd-1 tries to establish a connection to etcd-2, the following happens:
etcd-1presents itsetcd-1.crttoetcd-2.etcd-2uses its--trusted-ca-file(which is the sameca.crt) to verify thatetcd-1.crtwas indeed signed by the trusted CA.etcd-2then checks the Common Name (CN) inetcd-1.crt. It looks foretcd-1.example.comin its own--cert-allowed-cnlist. If it finds it, the connection is allowed.- Simultaneously,
etcd-2presents itsetcd-2.crttoetcd-1. etcd-1verifiesetcd-2.crtagainst itsca.crtand checks ifetcd-2.example.comis present in its--cert-allowed-cnlist.
This mutual authentication ensures that only authorized etcd peers can join the cluster.
The problem this solves is preventing rogue nodes from joining your etcd cluster and potentially corrupting data or eavesdropping on traffic. Without --cert-allowed-cn, a node could present a valid TLS certificate signed by your CA, but etcd wouldn’t know which node it was supposed to be. You’d be trusting any certificate signed by your CA, which is a significant security gap.
The --cert-allowed-cn flag is essential for enforcing which specific peers are allowed to connect. If you omit it, etcd will accept any peer certificate signed by the --trusted-ca-file, regardless of its Common Name. This means if an attacker compromises a machine and generates a certificate with a valid CN (e.g., etcd-3.example.com) signed by your CA, they could impersonate etcd-3 and join your cluster.
A common mistake is to forget to include all peer CNs in the --cert-allowed-cn list on every node. If etcd-1 has etcd-1.example.com and etcd-2.example.com in its list, but etcd-2 is missing etcd-1.example.com, then etcd-1 can talk to etcd-2, but etcd-2 will reject etcd-1’s connection. This leads to a split-brain scenario or a cluster that cannot form.
The CN in your certificate signing request (CSR) should ideally map directly to the --name of the etcd peer or a resolvable hostname for that peer. This makes the configuration more robust and easier to manage.
If you’ve configured --cert-allowed-cn correctly but still see connection issues, double-check that the peer-cert-file and peer-key-file paths are correct on each node and that the files are readable by the etcd process. Also, ensure the trusted-ca-file is identical across all nodes.
The next common configuration hurdle will be setting up client-to-etcd TLS authentication, which uses a similar but distinct set of flags.