Consul’s primary function is to provide a distributed, highly available service catalog and configuration store, but what truly makes it production-ready is its ability to securely and reliably manage that state across multiple machines and networks.
Let’s see this in action. Imagine we have three Consul servers (consul-01, consul-02, consul-03) and two Consul clients (consul-client-01, consul-client-02) running on a private network.
On each server, we’d start Consul with a basic configuration:
# /etc/consul.d/server.hcl
server = true
bootstrap_expect = 3
datacenter = "dc1"
data_dir = "/opt/consul/data"
log_level = "INFO"
client_addr = "0.0.0.0"
bind_addr = "192.168.1.101" # Unique IP for each server
And on the clients:
# /etc/consul.d/client.hcl
server = false
datacenter = "dc1"
data_dir = "/opt/consul/data"
log_level = "INFO"
client_addr = "0.0.0.0"
bind_addr = "192.168.1.105" # Unique IP for each client
To start a server node and join it to a cluster (assuming consul-01 is the first node):
consul agent -config-dir=/etc/consul.d -node="consul-01" -join="192.168.1.101"
And for subsequent servers (consul-02, consul-03):
consul agent -config-dir=/etc/consul.d -node="consul-02" -join="192.168.1.101"
consul agent -config-dir=/etc/consul.d -node="consul-03" -join="192.168.1.101"
Clients join similarly, pointing to one of the server’s addresses:
consul agent -config-dir=/etc/consul.d -node="consul-client-01" -join="192.168.1.101"
The bootstrap_expect setting on the servers tells Consul how many servers we expect to form the consensus quorum. When the first server starts, it’s in "bootstrap mode." Once bootstrap_expect nodes are running and reachable, Consul transitions out of bootstrap mode and establishes its Raft quorum. This prevents accidental split-brain scenarios where multiple independent clusters could form if servers are brought online too quickly without coordination.
Now, let’s layer on security. TLS is critical for encrypting communication between Consul agents and for verifying their identities.
First, generate a Certificate Authority (CA) and then server and client certificates signed by that CA. You’ll need openssl or a similar tool for this. The key is to ensure the certificates are valid for the IP addresses and hostnames Consul agents will use to communicate.
On each server and client, you’ll need a tls section in their configuration:
# /etc/consul.d/tls.hcl (merged with server.hcl or client.hcl)
tls {
enabled = true
cert_file = "/etc/consul.d/certs/consul.pem"
key_file = "/etc/consul.d/certs/consul-key.pem"
ca_file = "/etc/consul.d/certs/ca.pem"
}
When starting Consul agents with TLS enabled, you’d typically use:
consul agent -config-dir=/etc/consul.d -node="consul-01" -join="192.168.1.101" -tls-cert-file="/etc/consul.d/certs/consul-01.pem" -tls-key-file="/etc/consul.d/certs/consul-01-key.pem" -tls-ca-file="/etc/consul.d/certs/ca.pem"
This ensures all gossip traffic and API requests are encrypted and authenticated.
Finally, Access Control Lists (ACLs) provide fine-grained authorization. By default, Consul allows all operations. To secure it, you need to create an initial management token.
Start Consul with a bootstrap token set in your server configuration:
# /etc/consul.d/server.hcl
server = true
bootstrap_expect = 3
datacenter = "dc1"
data_dir = "/opt/consul/data"
log_level = "INFO"
client_addr = "0.0.0.0"
bind_addr = "192.168.1.101"
acl_datacenter = "dc1"
acl_default_policy = "deny"
acl_master_token = "my-super-secret-bootstrap-token"
After starting the cluster with this configuration, immediately create a new, more secure master token and disable the bootstrap token in the configuration. Use the bootstrap token to access the API:
export CONSUL_HTTP_TOKEN="my-super-secret-bootstrap-token"
consul acl bootstrap --name="initial-management" --format=json > /tmp/bootstrap_token.json
Then, extract the new SecretID from /tmp/bootstrap_token.json and use it to update your acl_master_token in the server configuration. Crucially, you must then remove the acl_master_token from the configuration file and restart the Consul servers. The token is only used for the initial bootstrapping of the ACL system.
# /etc/consul.d/server.hcl (after bootstrapping and restarting servers)
server = true
bootstrap_expect = 3
datacenter = "dc1"
data_dir = "/opt/consul/data"
log_level = "INFO"
client_addr = "0.0.0.0"
bind_addr = "192.168.1.101"
acl_datacenter = "dc1"
acl_default_policy = "deny"
# acl_master_token is REMOVED here
Now, you can use the new master token to create tokens for specific roles (e.g., read-only for services, write for service registration) and define policies that grant permissions to those tokens. For example, to create a token that can only read service information:
consul token create -name="service-reader" -policy="service-reader-policy" -format=json > /tmp/service_reader_token.json
The acl_default_policy = "deny" ensures that any request without an explicit token or a token with insufficient permissions is rejected. The acl_datacenter setting specifies which datacenter’s ACL table is authoritative.
The client_addr setting is crucial for allowing external connections to the Consul agent’s API and UI, while bind_addr is used for gossip and inter-node communication. If these are not correctly configured, nodes may fail to discover each other or clients may not be able to reach the servers.
The most surprising thing about Consul’s service discovery is that it doesn’t actually discover services; it relies on services actively registering themselves with Consul, or on external tools like consul-registrator or consul-template to perform this registration. Consul itself is merely the registry.
With a production-ready Consul cluster, the next logical step is to integrate your applications with it, using Consul’s health checks to ensure only healthy services are discovered and advertised.