A DestinationRule in Istio is your primary tool for configuring circuit breaking, but it’s not the only place you’ll interact with these protections.

Let’s see this in action. Imagine we have two services, productpage and reviews. The productpage service calls the reviews service. We want to protect productpage from a failing reviews service by implementing circuit breaking.

Here’s a sample DestinationRule that configures circuit breaking for the reviews service:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews-circuitbreaker
spec:
  host: reviews
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
        connectTimeout: 10ms
      http:
        http1MaxPendingRequests: 10
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
        maxConcurrentRequests: 100
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

This DestinationRule tells Istio how to behave when interacting with the reviews service.

The host field specifies which service this rule applies to. In this case, it’s reviews.

The trafficPolicy is where the circuit breaking configuration lives. It has two key sub-sections: connectionPool and outlierDetection.

connectionPool limits the number of connections and requests that can be made to the reviews service.

  • tcp.maxConnections: This sets a hard limit of 100 concurrent TCP connections to the reviews service. If this limit is reached, new connections will be queued or rejected depending on the connectTimeout.
  • tcp.connectTimeout: A 10ms timeout for establishing a TCP connection. If a connection can’t be established within this time, it’s considered a failure.
  • http.http1MaxPendingRequests: For HTTP/1.1, this limits the number of pending requests on a single connection. If this limit is reached, new requests are rejected.
  • http.http2MaxRequests: For HTTP/2, this limits the number of concurrent requests on a connection.
  • http.maxRequestsPerConnection: Limits the number of requests that can be sent over a single HTTP connection.
  • http.maxConcurrentRequests: This is a crucial setting. It limits the total number of concurrent requests (not connections) that Istio will allow to the reviews service at any given time. If this limit is hit, any new incoming requests will be rejected immediately.

outlierDetection is what actually implements the "tripping" of the circuit breaker. It defines conditions under which Istio will consider a reviews service instance "unhealthy" and temporarily stop sending traffic to it.

  • consecutive5xxErrors: If a reviews service instance returns 5 consecutive HTTP 5xx errors, Istio will eject it. This is the primary trigger for circuit breaking.
  • interval: After an instance is ejected, Istio will wait for 10 seconds before considering it for re-insertion.
  • baseEjectionTime: This is the minimum amount of time an instance will be ejected. In this case, it’s 30 seconds. If the ejection is caused by a high number of errors, the ejection time can be extended.
  • maxEjectionPercent: This limits the percentage of total reviews service instances that can be ejected at any one time. If 50% of the instances are unhealthy and ejected, Istio will stop ejecting more, even if the error threshold is met, to avoid a complete service outage.

When consecutive5xxErrors reaches 5 for a specific reviews pod, that pod will be temporarily removed from the load balancing pool for 30 seconds (or longer if errors persist). During this "ejection" period, Istio will return an 503 Service Unavailable error to the productpage service immediately, without even attempting to connect to the unhealthy reviews pod. This prevents the productpage service from wasting resources on a failing backend.

The maxConcurrentRequests in connectionPool works in tandem with outlierDetection. If the reviews service is slow but not returning errors, maxConcurrentRequests will prevent productpage from overwhelming it. Once a certain threshold of requests are failing (triggering outlierDetection), the circuit breaker will physically stop traffic.

The most surprising thing about outlierDetection is that it’s not just about HTTP errors. While consecutive5xxErrors is the most common trigger, Istio can also eject hosts based on TCP connection failures or timeouts. You can configure specific timeouts within connectionPool (like connectTimeout) that, if exceeded, can also contribute to a host being marked as an outlier, even if no explicit HTTP error code is returned.

This mechanism provides resilience by allowing the reviews service time to recover without being constantly hammered by requests that are likely to fail.

Once your circuit breakers are correctly configured and the reviews service is healthy again, you’ll likely want to explore more advanced traffic management features, such as request routing based on headers or fault injection for testing.

Want structured learning?

Take the full Circuit-breaker course →