Envoy’s connection pools are the unsung heroes of high-throughput services, but tuning them is less about magic and more about understanding the subtle dance between network latency and application processing time.
Let’s watch a real Envoy configuration in action, specifically how it handles upstream connections for a bustling microservice. Imagine service-a needs to talk to service-b. Envoy sits in front of service-b.
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
codec_type: AUTO
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/service-b"
route:
cluster: service_b_cluster
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: service_b_cluster
connect_timeout: 0.25s
type: LOGICAL_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_b_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 192.168.1.100
port_value: 8080
# Connection Pool Configuration Starts Here
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http3_protocol_options: {} # Example: Enabling HTTP/3
common_http_protocol_options:
max_requests_per_connection: 1000 # Default is often too low
idle_connection_timeout: 60s
# Other options like rate_limit_settings, etc.
This Envoy configuration defines a listener that accepts incoming HTTP traffic and routes requests prefixed with /service-b to service_b_cluster. The service_b_cluster points to an upstream service at 192.168.1.100:8080. The crucial part for connection pooling is within the typed_extension_protocol_options for the cluster.
The problem this solves is managing the overhead of establishing new TCP connections for every single request. For high-throughput services, this overhead can become a significant bottleneck. Envoy’s connection pooling reuses existing TCP connections to the upstream service, reducing latency and improving resource utilization on both the client (Envoy) and server sides.
Internally, Envoy maintains a pool of connections for each upstream cluster. When a request arrives, Envoy checks if an idle, reusable connection is available in the pool for the target upstream host. If so, it uses that connection. If not, and the pool isn’t full, it establishes a new connection. If the pool is full and no idle connections are available, the request might be queued or rejected depending on configuration.
The key levers you control are primarily within common_http_protocol_options and typed_extension_protocol_options:
max_requests_per_connection: This is the maximum number of requests that can be sent over a single persistent upstream connection before Envoy will close it and open a new one. A higher value means fewer connection setups but can lead to "stale" connections if the upstream has issues or if long-lived requests fill up the connection. For high-throughput, you’ll often want this quite high, like1000or even10000, allowing many requests to be multiplexed.idle_connection_timeout: This dictates how long an upstream connection can remain idle before Envoy closes it. A shorter timeout reduces the number of idle connections consuming resources but increases the churn (more frequent connection openings and closings). For services with consistent traffic, a longer timeout like60sor120scan be beneficial. For bursty traffic, a shorter timeout might be better to free up resources.max_connections: (Not shown in the example, but crucial) This defines the maximum number of connections Envoy will maintain to a single upstream host within a cluster. If this limit is reached, Envoy will not open new connections to that host, potentially leading to request queuing or rejection. This is your safety valve against overwhelming a specific upstream instance. Setting this too low starves your service; too high can overwhelm the upstream.concurrency: (Also not shown, but related) This setting on the listener level can limit the number of concurrent connections to Envoy itself. While not directly a connection pool setting for upstream, it impacts how many upstream requests Envoy can even generate in the first place.
The most overlooked aspect of connection pool tuning is how max_requests_per_connection interacts with upstream server behavior and network timeouts. If your upstream server has a request timeout (e.g., 30 seconds) and max_requests_per_connection is set to 10000, a single slow request can theoretically tie up that connection for a very long time, preventing other requests from using it, even if those other requests are fast. It’s a balancing act: you want to maximize reuse, but not at the expense of making the pool effectively smaller due to long-lived requests.
The next logical step after tuning connection pools is understanding how Envoy’s circuit breakers operate.