You can peer Consul clusters across different environments, like dev and prod, without ever touching WAN federation.
Let’s see it in action. Imagine you have a Consul cluster running in us-east-1 (your "primary") and another in eu-west-2 (your "secondary"). You want services registered in us-east-1 to be visible to services in eu-west-2, and vice-versa, but without the complexity of setting up Consul’s WAN federation.
Here’s a snapshot of a Consul agent configuration on the secondary cluster, showing how it connects to the primary:
# /etc/consul.d/consul.hcl (on a secondary cluster node)
server = false
client = true
# ... other client configurations ...
connect {
enabled = true
}
# This is the key part: pointing to the primary cluster's server address
# Use the actual IP or DNS name of a server in your primary cluster
consul {
address = "1.2.3.4:8500"
}
# This tells the secondary cluster to join the primary cluster's gossip pool
# It's not strictly necessary if you're only using the 'consul' address above
# for service discovery, but good for full cluster membership visibility.
# Replace with the actual IP and port of a server in the primary cluster.
# This is usually the same server you're pointing to in 'consul.address'
join = ["1.2.3.4"]
Now, on the primary cluster (let’s say its server is at 5.6.7.8), you’d configure it to accept incoming gossip from the secondary cluster’s IPs. This is often done implicitly if your security groups/firewalls allow it, but explicitly, you’d ensure the primary server’s firewall allows UDP traffic on port 8301 from the secondary cluster’s IPs. The primary server config doesn’t need a join pointing back unless you want bidirectional gossip membership.
The core idea here is that one cluster (the secondary) acts as a client to the other cluster (the primary) for its service discovery needs. It’s not a true peer-to-peer relationship in the WAN federation sense, but rather a directed connection. The secondary cluster’s agents query the primary cluster’s API (consul.address in the config) to resolve services registered in the primary.
Why would you do this instead of WAN federation? WAN federation is designed for geographically distributed, highly available, and resilient clusters that need to operate independently but share service information. It involves setting up trust, managing multiple server pools, and handling network partitions gracefully. It’s powerful but complex.
This "client-to-server" peering bypasses all of that. It’s simpler to set up because you’re just telling one cluster where to find the servers of another. It’s useful for:
- Development/Staging mirroring Production: A dev cluster can pull service definitions from production to test integrations without needing access to production infrastructure directly.
- Cross-environment visibility: A security monitoring tool in one environment might need to see services in another.
- Controlled information sharing: You can expose only specific services from one cluster to another by how you configure the clients.
The mechanism is straightforward: the consul.address configuration tells the Consul client agent on the secondary cluster to forward all its service discovery queries (like consul services or consul kv get) to the specified address on the primary cluster. If you also set join to an IP on the primary cluster, the secondary agent will also attempt to gossip with the primary cluster’s servers, making the primary servers aware of the secondary agent’s existence. This allows the primary cluster to potentially forward requests back to the secondary if needed, though the primary direction of service discovery is from secondary to primary.
Crucially, this doesn’t mean the servers of the secondary cluster are part of the primary cluster’s consensus or gossip pool. The secondary cluster is still an independent entity. Its agents are just configured to ask another cluster for information.
The most surprising thing about this approach is that it works for both service discovery and the Consul Connect service mesh. When a service on the secondary cluster wants to connect to a service on the primary cluster using Consul Connect, the secondary cluster’s Consul agent will query the primary cluster’s Consul API to get the latest service mesh configuration and endpoint information for the target service. The proxy on the secondary cluster can then establish the connection, potentially traversing networks if necessary.
This technique effectively creates a one-way or two-way "view" of services without the full overhead of a federated setup. It’s a pragmatic way to achieve cross-environment service awareness when strict multi-datacenter resilience isn’t the primary goal.
If you try to use consul services on the primary cluster and expect to see services registered only on the secondary cluster, you won’t. This setup primarily enables the secondary to discover services on the primary. For bidirectional discovery where both clusters see each other’s services automatically, you’d typically need WAN federation or a more complex setup involving multiple directed joins and careful network configuration.
The next problem you’ll likely encounter is managing network connectivity and TLS between these peered clusters, especially if they aren’t in the same VPC or cloud region.