Prometheus doesn’t actually collect metrics from your application; it scrapes them from an endpoint your application exposes.
Here’s how you can expose your application’s circuit breaker metrics in a Prometheus-friendly format. We’ll use the resilience4j library in Java as an example, as it’s a popular choice and has built-in support for Prometheus.
First, you need to add the resilience4j-prometheus dependency to your project. If you’re using Maven, it looks like this:
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-prometheus</artifactId>
<version>1.7.0</version> <!-- Use the latest version -->
</dependency>
Next, you need to configure your circuit breaker to register its metrics with a PrometheusRegistry. This registry will then expose these metrics via an HTTP endpoint.
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.github.resilience4j.circuitbreaker.CircuitBreakerRegistry;
import io.github.resilience4j.prometheus.PrometheusConfig;
import io.github.resilience4j.prometheus.PrometheusRegistry;
import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.exporter.HTTPServer;
import java.io.IOException;
import java.time.Duration;
public class CircuitBreakerPrometheusExporter {
public static void main(String[] args) throws IOException {
// 1. Create a Prometheus Registry
// This is a global registry that Prometheus will scrape from.
PrometheusRegistry registry = new PrometheusRegistry(PrometheusConfig.DEFAULT);
// 2. Configure your Circuit Breaker
// Standard resilience4j configuration.
CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerConfig.custom()
.failureRateThreshold(50) // If 50% of calls fail, open the circuit
.waitDurationInOpenState(Duration.ofMinutes(1)) // Stay open for 1 minute
.slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
.slidingWindowSize(10) // Consider the last 10 calls
.build();
// 3. Create a Circuit Breaker Registry
// This registry holds all your circuit breakers.
CircuitBreakerRegistry circuitBreakerRegistry = CircuitBreakerRegistry.of(circuitBreakerConfig);
// 4. Register the Circuit Breaker with the Prometheus Registry
// This step is crucial. It tells the Prometheus registry to collect metrics
// from this specific circuit breaker.
CircuitBreaker circuitBreaker = circuitBreakerRegistry.circuitBreaker("myService");
registry.registerCircuitBreaker(circuitBreaker);
// 5. Start an HTTP Server to expose metrics
// Prometheus will scrape this endpoint. The default port is 9091.
// You can configure this port or host if needed.
int port = 9091;
HTTPServer server = new HTTPServer(port);
System.out.println("Prometheus metrics exposed on http://localhost:" + port + "/metrics");
// --- Simulate some calls to the circuit breaker ---
// In a real application, you'd wrap your service calls with this.
System.out.println("Simulating calls...");
for (int i = 0; i < 20; i++) {
try {
// Simulate a successful call
if (i % 3 == 0) {
circuitBreaker.onSuccess(Duration.ofMillis(100));
System.out.println("Call " + i + ": Success");
} else {
// Simulate a failed call
circuitBreaker.onError(Duration.ofMillis(100), new RuntimeException("Simulated error"));
System.out.println("Call " + i + ": Error");
}
} catch (Exception e) {
// This catch block is for when the circuit is open and calls are blocked.
System.out.println("Call " + i + ": Circuit breaker is open, call blocked.");
}
Thread.sleep(500); // Small delay between calls
}
// Keep the application running to allow Prometheus to scrape
// In a real application, this would be your main application thread.
// Thread.currentThread().join();
}
}
When you run this, a Prometheus-compatible metrics endpoint will be available at http://localhost:9091/metrics. You’ll see metrics like these:
# HELP circuit_breaker_calls_total Total number of calls made to the circuit breaker.
# TYPE circuit_breaker_calls_total counter
circuit_breaker_calls_total{name="myService",kind="SUCCESS"} 7.0
circuit_breaker_calls_total{name="myService",kind="ERROR"} 13.0
# HELP circuit_breaker_state Current state of the circuit breaker.
# TYPE circuit_breaker_state gauge
circuit_breaker_state{name="myService",state="CLOSED"} 1.0
# HELP circuit_breaker_state_changes_total Total number of state changes.
# TYPE circuit_breaker_state_changes_total counter
circuit_breaker_state_changes_total{name="myService",from="CLOSED",to="OPEN"} 1.0
To actually see these metrics in Prometheus, you need to configure Prometheus to scrape this endpoint. In your prometheus.yml configuration, you’d add a scrape job:
scrape_configs:
- job_name: 'my_app'
static_configs:
- targets: ['localhost:9091'] # The host and port where your application exposes metrics
After restarting Prometheus and ensuring your application is running, you can go to the Prometheus UI and query metrics like circuit_breaker_calls_total{job="my_app", name="myService"}.
The truly surprising thing is how little Prometheus does by default; it’s a pull-based system that relies entirely on your applications to expose data in a specific format.
The most counterintuitive aspect of circuit breaker metrics is that the circuit_breaker_state metric is a gauge that will show 1.0 for the current state and 0.0 for others. So, if the breaker is CLOSED, circuit_breaker_state{name="myService",state="CLOSED"} will be 1.0, while circuit_breaker_state{name="myService",state="OPEN"} and circuit_breaker_state{name="myService",state="HALF_OPEN"} will be 0.0. This allows you to easily alert on state changes.
The next step is to visualize these metrics in Grafana dashboards.