Envoy’s dynamic configuration, powered by the Admin API and xDS gRPC services, means you’re not just configuring a proxy; you’re orchestrating a distributed system where services discover and adapt to each other in real-time.
Let’s see this in action. Imagine you have a simple nginx service running on port 8080. We want Envoy to route traffic to it, but we want to be able to update the routing rules without restarting Envoy.
First, we need a management server. For this example, we’ll use a mock management server that serves static configuration. In a real-world scenario, this would be a sophisticated system like Istio’s Pilot or a custom solution.
Here’s a minimal Envoy configuration:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: service_a
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: service_a
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: service_a
connect_timeout: 0.25s
type: LOGICAL_DNS
lb_policy: ROUND_ROBIN
dns_lookup_family: V4_ONLY
# This is where we'd point to our dynamic discovery service.
# For simplicity, we'll use a static cluster for now to show Envoy running.
# In a real ADS setup, this would be absent or point to a bootstrap config.
load_assignment:
cluster_name: service_a
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
admin:
access_log_path: /tmp/envoy.log
address:
socket_address:
address: 127.0.0.1
port_value: 9901
Now, let’s set up Envoy to use ADS. The key is the dynamic_resources section in the bootstrap configuration. Instead of static_resources, we’ll have ads_config.
Here’s the bootstrap configuration for dynamic configuration:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
# Route configuration will be dynamically loaded via ADS
# Initially, this might be empty or point to a default route.
route_config:
name: local_route
virtual_hosts: [] # Empty for dynamic loading
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters: # Clusters will also be dynamically loaded via ADS
- name: ads_cluster # This cluster points to the management server
connect_timeout: 1.0s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: ads_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1 # Address of your xDS server
port_value: 50001 # Port of your xDS server
admin:
access_log_path: /tmp/envoy.log
address:
socket_address:
address: 127.0.0.1
port_value: 9901
dynamic_resources:
ads_config:
api_type: GRPC
grpc_services:
- envoy_cluster_name: "ads_cluster" # This cluster name must match a cluster defined above
timeout: 0.25s
# The following specify which xDS resources to fetch.
# For HTTP traffic, we primarily care about RDS, CDS, and EDS.
# Listeners are configured statically in this bootstrap.
lds_config: # Listener Discovery Service
ads: {}
cds_config: # Cluster Discovery Service
ads: {}
eds_config: # Endpoint Discovery Service
ads: {}
# For route configuration, we need to specify Route Configuration Discovery Service (RTDS)
# This is typically enabled via a listener filter or by specifying it in the HttpConnectionManager config.
# In this bootstrap, we've left route_config empty, implying RTDS will populate it.
# The actual RTDS config is usually part of the LDS response.
With this setup, Envoy starts, connects to 127.0.0.1:50001 (our hypothetical xDS server), and begins requesting Listener, Cluster, and Endpoint configurations. The ads_config block tells Envoy how to connect to the xDS server and which resources to ask for. The grpc_services list is crucial here, specifying the envoy_cluster_name that resolves to the xDS server’s address.
The lds_config, cds_config, and eds_config set to ads: {} instruct Envoy to use the ADS gRPC stream for these resource types. This means Envoy will establish a persistent gRPC connection to the management server and subscribe to updates for listeners, clusters, and endpoints. The management server then pushes configuration changes down these streams.
Let’s simulate a management server response. Suppose our management server pushes the following Listener configuration:
{
"name": "listener_0",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 10000
}
},
"filter_chains": [
{
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"stat_prefix": "ingress_http",
"route_config": {
"name": "local_route",
"virtual_hosts": [
{
"name": "service_a_vh",
"domains": ["*"],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "service_a"
}
}
]
}
]
},
"http_filters": [
{
"name": "envoy.filters.http.router",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
}
}
]
}
}
]
}
]
}
And a Cluster configuration:
{
"name": "service_a",
"connect_timeout": "0.25s",
"type": "STRICT_DNS", // Changed to STRICT_DNS for dynamic updates
"lb_policy": "ROUND_ROBIN",
"dns_lookup_family": "V4_ONLY",
"load_assignment": {
"cluster_name": "service_a",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 8080
}
}
}
}
]
}
]
}
}
Once Envoy receives these, it updates its configuration without restarting. You can verify this by checking the Envoy Admin API (curl localhost:9901/clusters and curl localhost:9901/listeners).
The real power of ADS is its ability to push updates. If we change the upstream service address for service_a to 127.0.0.1:8081 and push a new Cluster configuration, Envoy will seamlessly switch traffic without interruption. This is because the gRPC streams are long-lived and Envoy continuously watches for changes.
The management server can also push Route Configuration Discovery Service (RTDS) updates. This allows dynamic updates to the routing rules within a Virtual Host. Instead of embedding route_config directly in the Listener’s HttpConnectionManager (as shown above), you’d typically have an empty route_config and a rds field pointing to an api_config that specifies the RTDS server. Envoy would then fetch route configurations dynamically.
When Envoy is configured with ads_config, it essentially becomes a client to your xDS server. The ads_cluster is the Envoy cluster that defines how to reach your xDS server. The grpc_services within ads_config tell Envoy which xDS APIs (LDS, CDS, EDS, RTDS) to use via that ads_cluster. The ads: {} syntax indicates that the ADS gRPC stream should be used for that specific discovery type.
The trick to making this work without a full-blown management server for testing is to use Envoy’s xds-relay or a simple gRPC server that implements the xDS API. You can also configure Envoy to fetch initial configuration from a static file and then switch to ADS, or vice-versa. The dynamic_resources block in the bootstrap config is the primary switch.
The underlying mechanism is a set of persistent gRPC bidirectional streams. Envoy initiates these streams to the management server for each xDS resource type (Listeners, Clusters, Endpoints, Routes). The management server then uses these streams to push configuration updates to Envoy. This is a publish-subscribe model where Envoy subscribes to configuration changes.
When you update a cluster’s endpoint address, Envoy doesn’t just swap configuration; it updates its internal connection pool and load balancing state for that cluster. For listeners, it might update routing tables or add/remove filters. The key is that these updates are applied hot, meaning no packet loss or downtime if done correctly.
The next step in mastering Envoy’s dynamic configuration is understanding how to manage secrets and certificates using SDS (Secret Discovery Service), which also integrates seamlessly with ADS.