When a service you depend on starts throwing errors, the default reaction is to panic and start debugging the failing service. But what if you could serve something useful from your own service, even when that dependency is down?
Let’s see this in action. Imagine we have a product-service that relies on inventory-service to check stock levels before fulfilling an order.
@Service
public class ProductService {
private final InventoryServiceClient inventoryServiceClient;
private final CircuitBreaker circuitBreaker; // Assume this is injected
public ProductService(InventoryServiceClient inventoryServiceClient, CircuitBreakerFactory circuitBreakerFactory) {
this.inventoryServiceClient = inventoryServiceClient;
this.circuitBreaker = circuitBreakerFactory.create("inventoryCircuitBreaker");
}
public Product getProductWithStock(String productId) {
// Attempt to get stock from inventory service
StockLevel stock = circuitBreaker.run(
() -> inventoryServiceClient.getStockLevel(productId), // The actual call
throwable -> new StockLevel(productId, 0) // Fallback logic
);
// ... construct Product object using stock ...
return new Product(productId, "Example Product", stock.getLevel());
}
}
Here, InventoryServiceClient is a Feign client or similar, and circuitBreaker.run is the core of our strategy. The first lambda () -> inventoryServiceClient.getStockLevel(productId) is the "happy path" call. The second lambda throwable -> new StockLevel(productId, 0) is the fallback. If the inventoryServiceClient.getStockLevel call fails (throws an exception) or times out, the circuitBreaker will execute the fallback, returning a StockLevel with 0.
This is the fundamental mechanism: a circuit breaker wraps a potentially failing operation. If the operation fails too many times in a row, the circuit breaker "opens," and subsequent calls immediately execute a fallback instead of attempting the failing operation. After a timeout, it enters a "half-open" state, allowing a single test call; if that succeeds, it closes, otherwise, it opens again.
The problem this solves is cascading failures. Without a fallback, if inventory-service is slow or down, product-service will also become slow or down, and any service depending on product-service will also fail. By serving a cached response or a default value, product-service can remain partially functional, providing a degraded but still useful experience.
The key levers you control are:
- Failure Threshold: How many failures trigger the circuit breaker to open. (e.g., 5 consecutive failures)
- Failure Rate Threshold: The percentage of calls that must fail to open the circuit. (e.g., 50% of calls in a sliding window)
- Wait Duration: How long the circuit stays open before attempting a half-open state. (e.g., 30 seconds)
- Fallback Logic: What to return when the circuit is open. This is where serving cached data comes in.
Your fallback doesn’t have to be a hardcoded default like 0 stock. It can be a value retrieved from a local cache.
@Service
public class ProductService {
private final InventoryServiceClient inventoryServiceClient;
private final Cache<String, StockLevel> stockCache; // Assume this is a Caffeine or Guava cache
private final CircuitBreaker circuitBreaker;
public ProductService(InventoryServiceClient inventoryServiceClient, CacheFactory cacheFactory, CircuitBreakerFactory circuitBreakerFactory) {
this.inventoryServiceClient = inventoryServiceClient;
this.stockCache = cacheFactory.createStockCache(); // e.g., 5 minutes TTL
this.circuitBreaker = circuitBreakerFactory.create("inventoryCircuitBreaker");
}
public Product getProductWithStock(String productId) {
StockLevel stock = circuitBreaker.run(
() -> {
StockLevel fetchedStock = inventoryServiceClient.getStockLevel(productId);
stockCache.put(productId, fetchedStock); // Update cache on success
return fetchedStock;
},
throwable -> {
// Fallback: Try to get from cache
StockLevel cachedStock = stockCache.getIfPresent(productId);
if (cachedStock != null) {
return cachedStock; // Serve stale data from cache
}
return new StockLevel(productId, 0); // Last resort: default value
}
);
// ... construct Product object using stock ...
return new Product(productId, "Example Product", stock.getLevel());
}
}
In this enhanced example, when the inventoryServiceClient fails and the circuit breaker is open, we first check our local stockCache. If we have a stale StockLevel there, we serve that. This means your users might see an "out of stock" message for a product that’s actually available, or see an "in stock" message for a product that’s sold out, but they will still see a product. The cache is populated on successful calls to inventoryServiceClient.
The real magic is that the circuit breaker doesn’t just prevent calls when it’s open; it also manages the recovery. When the waitDuration expires, it allows one request to pass through to the inventoryServiceClient. If that single request succeeds, the circuit breaker transitions to a "half-open" state and allows more requests, effectively resuming normal operation. If it fails again, it immediately re-opens. This prevents a partially failed dependency from bringing your entire system down indefinitely.
When you configure your circuit breaker, you’re essentially tuning its sensitivity and resilience. For instance, using Resilience4j in Spring Boot, you might have configuration like this:
resilience4j.circuitbreaker:
instances:
inventoryCircuitBreaker:
registerHealthIndicator: true
slidingWindowType: COUNT_BASED
slidingWindowSize: 10
minimumNumberOfCalls: 5
permittedNumberOfCallsInHalfOpenState: 2
automaticTransitionFromOpenToHalfOpenEnabled: true
waitDurationInOpenState: 15s
recordExceptions:
- org.springframework.web.client.HttpServerErrorException
- java.util.concurrent.TimeoutException
This configuration states that the inventoryCircuitBreaker will open if, within the last 10 calls, at least 5 calls failed (or were recorded exceptions). While open, it will wait 15 seconds before allowing 2 calls through in a half-open state. If those 2 succeed, it closes.
The most surprising thing about this pattern is how often the fallback data is "good enough." Users rarely notice a slight staleness in inventory counts or product details compared to a complete failure. The system feels more robust and available.
The next thing you’ll likely encounter is how to manage cache invalidation when the circuit breaker is closed and the dependency is healthy.