Circuit breakers are a fundamental pattern for building resilient distributed systems, but understanding how to test them effectively across different scopes is crucial.
Let’s see a simple Go application that uses a circuit breaker to protect a potentially flaky downstream service.
package main
import (
"errors"
"fmt"
"log"
"net/http"
"time"
"github.com/sony/gobreaker"
)
// Mock downstream service
func downstreamService(w http.ResponseWriter, r *http.Request) {
// Simulate occasional failures
if time.Now().Second()%3 == 0 {
http.Error(w, "Service unavailable", http.StatusInternalServerError)
return
}
fmt.Fprintln(w, "Service healthy")
}
// Circuit breaker configuration
var breaker *gobreaker.CircuitBreaker
func init() {
settings := gobreaker.Settings{
Name: "downstream-service-breaker",
MaxRequests: 3, // Number of requests allowed when in HALF-OPEN state
Timeout: 10 * time.Second, // How long the breaker stays OPEN before transitioning to HALF-OPEN
ReadyToTrip: func(counts gobreaker.Counts) bool {
// Trip the breaker if 50% of requests fail
failureRate := float64(counts.Failure) / float64(counts.Total)
return failureRate >= 0.5 && counts.Total >= 3
},
OnStateChange: func(name string, from, to gobreaker.State) {
log.Printf("Circuit breaker '%s' changed state from %s to %s\n", name, from, to)
},
}
breaker = gobreaker.NewCircuitBreaker(settings)
}
func callDownstream(w http.ResponseWriter, r *http.Request) {
operation := func() (interface{}, error) {
resp, err := http.Get("http://localhost:8081/downstream") // Assuming downstream runs on 8081
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, errors.New(fmt.Sprintf("unexpected status code: %d", resp.StatusCode))
}
return "Success", nil
}
_, err := breaker.Execute(operation)
if err != nil {
http.Error(w, fmt.Sprintf("Circuit breaker error: %v", err), http.StatusInternalServerError)
return
}
fmt.Fprintln(w, "Successfully called downstream service.")
}
func main() {
// Start the mock downstream service
go func() {
http.HandleFunc("/downstream", downstreamService)
log.Println("Starting downstream service on :8081")
if err := http.ListenAndServe(":8081", nil); err != nil {
log.Fatalf("Failed to start downstream service: %v", err)
}
}()
// Start the main application with the circuit breaker
http.HandleFunc("/call", callDownstream)
log.Println("Starting main application on :8080")
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Fatalf("Failed to start main application: %v", err)
}
}
This main function sets up two HTTP servers: one for the "downstream service" (which simulates failures) and one for the "main application" that calls the downstream service through a gobreaker circuit breaker. The breaker is configured to trip if 50% of requests fail (with a minimum of 3 total requests), and it stays open for 10 seconds.
The core problem circuit breakers solve is preventing cascading failures. When a service becomes unhealthy, repeatedly hammering it with requests will only worsen its state and potentially bring down other services that depend on it. A circuit breaker acts as a protective proxy, quickly failing requests for a period when it detects sustained errors, giving the downstream service time to recover and preventing your own service from wasting resources.
The gobreaker library implements the standard states: CLOSED, OPEN, and HALF-OPEN.
- CLOSED: The normal state. Requests are passed through to the downstream service. If failures exceed the
ReadyToTripthreshold, the breaker transitions to OPEN. - OPEN: Requests are immediately rejected with a
gobreaker.ErrOpenStateerror. After theTimeoutduration, the breaker transitions to HALF-OPEN. - HALF-OPEN: A limited number of requests (
MaxRequests) are allowed through. If these requests succeed, the breaker transitions back to CLOSED. If any fail, it immediately returns to OPEN.
The breaker.Execute(operation) method is the heart of it. It wraps your actual service call (operation) and handles the state transitions automatically.
The most surprising true thing about circuit breakers is that their primary benefit isn’t just preventing failures, but managing them. They turn hard failures into predictable, fast failures, allowing your system to degrade gracefully rather than collapse.
The ReadyToTrip function is where you define what constitutes "enough" failures to warrant opening the circuit. A simple count of failures isn’t always sufficient; you often want to consider the failure rate, as shown in the example (failureRate >= 0.5 && counts.Total >= 3). This prevents a single transient error from immediately tripping the breaker.
When you’re testing this system, you’ll want to simulate various scenarios:
- Successful calls: Ensure the breaker stays CLOSED.
- Intermittent failures: Trigger the
ReadyToTripcondition to move to OPEN. - Timeout: Wait for the breaker to go OPEN, then wait for the
Timeoutduration to pass, and observe the HALF-OPEN state. - Successful recovery in HALF-OPEN: Make a request in HALF-OPEN that succeeds, returning to CLOSED.
- Failure in HALF-OPEN: Make a request in HALF-OPEN that fails, returning to OPEN.
- Immediate rejection in OPEN: Attempt calls while the breaker is OPEN and verify they are rejected immediately.
This isn’t just about testing the circuit breaker logic itself, but how your application behaves when the circuit breaker is in different states. You’d typically use tools like curl or a dedicated testing framework to send requests to your application’s /call endpoint while manipulating the downstream service’s availability.
For instance, to simulate a failure, you might stop the downstream service entirely, or modify its downstreamService function to always return an error. To simulate recovery, you’d restart the downstream service or revert its function.
The OnStateChange callback is invaluable for debugging and understanding the breaker’s lifecycle during tests. It logs every transition, giving you a clear audit trail of what the breaker is doing.
The most common mistake when implementing circuit breakers is not having a clear strategy for what happens when the breaker is OPEN or HALF-OPEN. Your application shouldn’t just return a generic error; it should ideally have a fallback mechanism, like returning cached data or a default response, to maintain some level of functionality.
The next challenge you’ll face is orchestrating these circuit breaker states across multiple services in a complex microservices architecture.