Circuit breakers don’t just prevent cascading failures; they actively optimize for them by creating intentional, controlled outages.
Let’s watch a real-time circuit breaker in action. Imagine a service, user-service, that depends on order-service. If order-service starts to falter, user-service doesn’t want to keep hammering it and bringing itself down too.
Here’s a simplified user-service client configuration using Hystrix (a popular Java circuit breaker library, though the concepts apply universally):
// Configure the circuit breaker for calls to order-service
HystrixCommandProperties.Setter commandPropertiesDefaults = HystrixCommandProperties.Setter()
.withExecutionTimeoutInMilliseconds(1000) // How long to wait for a successful response
.withCircuitBreakerEnabled(true)
.withCircuitBreakerRequestVolumeThreshold(20) // Minimum requests before checking the breaker
.withCircuitBreakerSleepWindowInMilliseconds(5000) // How long to wait before trying again after tripping
.withCircuitBreakerErrorThresholdPercentage(50); // Percentage of errors to trip the breaker
// Create a command to call order-service
public class GetOrderCommand extends HystrixCommand<Order> {
private final String orderId;
public GetOrderCommand(String orderId) {
super(HystrixCommandGroupKey.Factory.asKey("OrderService"),
HystrixCommandKey.Factory.asKey("GetOrder"),
HystrixCommandProperties.Setter.from(commandPropertiesDefaults)); // Apply our default settings
this.orderId = orderId;
}
@Override
protected Order run() throws Exception {
// Actual call to order-service
return orderServiceHttpClient.getOrder(orderId);
}
@Override
protected Order getFallback() {
// What to do when the breaker is open or an error occurs
return new Order("fallback_order_" + orderId);
}
}
// In user-service's request handling:
String orderId = getOrderIdFromRequest();
Order order = new GetOrderCommand(orderId).execute();
// Process the order, using a fallback if necessary
This setup defines a few key parameters:
executionTimeoutInMilliseconds: Iforder-servicedoesn’t respond within 1 second, consider it a failure.circuitBreakerRequestVolumeThreshold: We need at least 20 requests toorder-servicewithin a rolling window to even consider tripping the breaker. This prevents a single transient glitch from shutting down healthy operations.circuitBreakerErrorThresholdPercentage: If 50% of those 20 requests fail (timeout, network error, explicit error response), the circuit breaker trips.circuitBreakerSleepWindowInMilliseconds: Once tripped, the breaker stays open for 5 seconds. During this time, all calls toorder-servicewill immediately return a fallback, bypassing the actual network call. This givesorder-servicea chance to recover. After 5 seconds, the breaker allows a single "test" request through. If that succeeds, the breaker closes; otherwise, it stays open.
The core problem circuit breakers solve is cascading failure. Without them, if order-service becomes slow or unavailable, user-service clients will pile up waiting for responses. These waiting threads consume resources (memory, CPU) on the user-service instances. Eventually, user-service itself runs out of resources and becomes unresponsive, impacting its clients, and so on. The circuit breaker acts as a shock absorber, intentionally failing fast and gracefully when a dependency is unhealthy, preserving the health of the calling service and allowing the dependency time to recover.
Tuning these thresholds is about finding the sweet spot between being too sensitive and not sensitive enough. If your SLA (Service Level Agreement) guarantees 99.9% availability for user-service, and order-service is a critical dependency, you can’t afford to let a struggling order-service drag user-service down.
- Low
circuitBreakerRequestVolumeThreshold: Too sensitive. A few random network blips might cause the breaker to trip unnecessarily, impacting performance for a brief period. - High
circuitBreakerRequestVolumeThreshold: Not sensitive enough. Many requests might be sent to a failing service before the breaker even considers tripping, leading to more widespread resource exhaustion. - Low
circuitBreakerErrorThresholdPercentage: Trips too easily. A temporary spike in errors, even if well below the expected failure rate for the SLA, could open the circuit. - High
circuitBreakerErrorThresholdPercentage: Too lenient. The service can be significantly degraded before the breaker intervenes, leading to poor user experience and potential cascading failures. - Short
circuitBreakerSleepWindowInMilliseconds: The dependency might not have enough time to recover before the breaker attempts to close, leading to rapid tripping and re-opening. - Long
circuitBreakerSleepWindowInMilliseconds: Users might experience fallback behavior for longer than necessary if the dependency recovers quickly.
The key is to align these values with your SLA and the observed behavior of your dependencies. If your SLA states order-service must respond within 500ms 99.9% of the time, and user-service experiences timeouts when order-service exceeds that, you’d set executionTimeoutInMilliseconds to something like 600ms. You’d then monitor the error rate. If you see more than 0.1% of requests to order-service failing (timing out or returning errors) over a given window, that’s your signal to start tuning circuitBreakerErrorThresholdPercentage. A common starting point for a 99.9% availability SLA is to set the error threshold to something like 0.1% or 0.5% of requests, but this requires careful monitoring and adjustment. The circuitBreakerRequestVolumeThreshold should be high enough to capture a statistically significant sample of requests to avoid noise, perhaps a few minutes worth of traffic. The circuitBreakerSleepWindowInMilliseconds should be long enough to allow for recovery but short enough to minimize user impact, often set to 10-30 seconds.
When you finally get the circuit breaker thresholds tuned perfectly, the next thing you’ll notice is how much harder it is to debug intermittent network latency issues.