You can think of circuit breaking as a fancy way to stop cascading failures in distributed systems, but the truly mind-bending part is that it’s often more effective when the service itself doesn’t know it’s happening.
Let’s see it in action. Imagine two services, frontend and backend. The frontend makes calls to backend. We want to prevent the frontend from hammering a backend that’s struggling.
Here’s a simplified backend service (Node.js, but the concept is universal):
const express = require('express');
const app = express();
const port = 3000;
// Simulate a failing backend
let failureRate = 0.0;
let invocations = 0;
app.get('/data', (req, res) => {
invocations++;
if (Math.random() < failureRate) {
console.log(`[Backend] Simulating failure for invocation ${invocations}`);
return res.status(503).send('Service Unavailable');
}
console.log(`[Backend] Successful call for invocation ${invocations}`);
res.send('Here is your data!');
});
// Endpoint to control failure rate for demonstration
app.post('/set-failure-rate', express.json(), (req, res) => {
const { rate } = req.body;
if (rate !== undefined) {
failureRate = parseFloat(rate);
console.log(`[Backend] Failure rate set to ${failureRate}`);
res.status(200).send(`Failure rate set to ${failureRate}`);
} else {
res.status(400).send('Missing rate parameter');
}
});
app.listen(port, () => {
console.log(`Backend service listening on port ${port}`);
});
And a frontend service that calls it:
const express = require('express');
const axios = require('axios');
const app = express();
const port = 8080;
const BACKEND_URL = 'http://localhost:3000/data'; // Assuming backend is on port 3000
app.get('/fetch-data', async (req, res) => {
try {
console.log('[Frontend] Calling backend...');
const response = await axios.get(BACKEND_URL, { timeout: 5000 }); // 5-second timeout
res.send(`Backend responded: ${response.data}`);
} catch (error) {
console.error('[Frontend] Error calling backend:', error.message);
res.status(500).send(`Failed to fetch data: ${error.message}`);
}
});
app.listen(port, () => {
console.log(`Frontend service listening on port ${port}`);
});
If you run these two and then tell the backend to fail often (e.g., curl -X POST -H "Content-Type: application/json" -d '{"rate": 0.8}' http://localhost:3000/set-failure-rate), the frontend logs will quickly fill with errors. The frontend keeps retrying, eventually timing out, but it’s still sending requests to a broken service.
Now, let’s introduce a service mesh like Istio. We’ll deploy our frontend and backend as Kubernetes pods, and have Istio’s istio-proxy (a sidecar) manage their network traffic.
First, we need Istio installed. The istio-proxy container will be automatically injected into our pods.
Here’s how we’d configure the circuit breaker on the service mesh, not the frontend application. This is a Kubernetes VirtualService and DestinationRule for Istio:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: backend-dr
spec:
host: backend # The Kubernetes service name for our backend
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100 # Limit concurrent TCP connections
http:
http1MaxPendingRequests: 10 # Max pending HTTP requests in buffer
http2MaxRequests: 100 # Max concurrent HTTP/2 requests
maxRequestsPerConnection: 1 # Important: Limit requests per TCP connection
outlierDetection:
consecutive5xxErrors: 3 # If 3 consecutive 5xx errors occur...
interval: 10s # ...check every 10 seconds...
baseEjectionTime: 60s # ...eject the pod for 60 seconds.
maxEjectionPercent: 50 # Don't eject more than 50% of pods.
# We can also add timeouts here, but often they are set in VirtualService
# The default timeout for Istio is 15s if not specified elsewhere.
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: frontend-vs
spec:
hosts:
- frontend # The Kubernetes service name for our frontend
http:
- route:
- destination:
host: backend # The Kubernetes service name for our backend
port:
number: 3000 # The port the backend service listens on
timeout: 5s # Explicitly set the request timeout (e.g., 5 seconds)
# In Istio, circuit breaking is primarily configured in DestinationRule's outlierDetection.
# The VirtualService defines routing, timeouts, and can specify retries.
# For explicit rate limiting or more advanced breaking, you'd use Istio's RateLimitService.
With this in place, the istio-proxy sidecar for the frontend pod is watching the responses from the backend service. When it sees 3 consecutive 5xx errors from a particular backend pod, it marks that pod as "unhealthy" and temporarily stops sending traffic to it for 60 seconds (baseEjectionTime).
The frontend application itself doesn’t need to know about circuit breakers. It just makes its HTTP calls. The istio-proxy intercepts these calls before they leave the frontend pod and after they arrive at the backend pod. If a backend pod is ejected, the istio-proxy on the frontend side will immediately return a 503 Service Unavailable (or a similar error) without even attempting to send the request. This is the core of offloading: the mesh handles the fault tolerance logic.
This dramatically reduces the load on the struggling backend service. Instead of the frontend application’s timeout logic eventually kicking in after a long, painful wait, the istio-proxy provides an almost instantaneous rejection for a period, giving the backend service a chance to recover. The maxRequestsPerConnection: 1 in the DestinationRule further helps by preventing a single slow connection from holding up resources.
The most surprising aspect of this is how this outlierDetection mechanism, by default, does not use a sophisticated algorithm to determine "unhealthiness." It’s a simple, albeit effective, counter for consecutive errors. A more nuanced approach might involve exponentially weighted moving averages or latency-based detection, but Istio’s default is a pragmatic starting point that solves many common failure scenarios by simply giving a failing instance a timeout.
The next challenge you’ll likely encounter is understanding how to implement sophisticated retry strategies in conjunction with circuit breaking to ensure graceful degradation rather than outright failure.