Caddy’s tracing capabilities aren’t about just seeing requests; they’re about seeing the journey of a request through Caddy’s internal middleware pipeline.
Let’s watch a request flow through Caddy with tracing enabled. Imagine we have a simple Caddyfile:
:8080 {
reverse_proxy http://localhost:8081
}
When a request hits Caddy on port 8080, Caddy starts a trace. The first span is for the overall request. If reverse_proxy is enabled, Caddy then initiates a new span for the reverse_proxy operation itself. If that reverse_proxy then hits another Caddy instance (or any other service instrumented for tracing), that downstream service will see the incoming trace context and continue the trace, creating its own spans. This is how you get end-to-end visibility.
Here’s how you’d enable it in your Caddyfile:
:8080 {
reverse_proxy http://localhost:8081
# Enable OpenTelemetry tracing
trace {
# Specify the exporter, e.g., OTLP to send traces to a collector
otlp {
# Endpoint of your OpenTelemetry Collector
endpoint 0.0.0.1:4317
# Optional: Use TLS for secure transport
tls
}
# Optionally, set a service name for your Caddy instance
service_name my-caddy-service
}
}
In this configuration:
trace {}: This block activates Caddy’s tracing module.otlp {}: This specifies the OpenTelemetry Protocol (OTLP) exporter. This is the most common way to send traces to an OpenTelemetry Collector, which then forwards them to a backend like Jaeger, Prometheus, or Honeycomb.endpoint 0.0.0.1:4317: This is the address where your OTLP collector is listening for traces. For local testing, this might belocalhost:4317or0.0.0.1:4317. In a production environment, it would be the address of your collector service.tls: If your collector uses TLS (which it should for production), you’d include this. Caddy will attempt to use the system’s default trust store to verify the collector’s certificate.service_name my-caddy-service: This tag is added to all spans originating from this Caddy instance, allowing you to filter and group traces by the originating service.
When a request arrives, Caddy generates a trace ID and a span ID for the initial request handling. It then propagates this trace context (usually via HTTP headers like traceparent and tracestate) to any downstream services it proxies to. If the downstream service is also instrumented, it will pick up this context, create a new span that’s a child of the Caddy span, and continue the trace.
The real power comes when you chain these. Imagine Caddy proxies to a microservice, which in turn proxies to another. With tracing enabled on all hops, you can see the latency introduced at each step of the request’s journey. You’re not just seeing that the overall request took 500ms; you’re seeing that Caddy took 50ms, the first microservice took 200ms, and the second microservice took 150ms, with 100ms of that being network latency between them.
This allows for pinpointing performance bottlenecks. If a request suddenly slows down, you can look at the trace and immediately see which component or hop is responsible for the increased latency. You can also see errors. If a downstream service returns an error, the span for that operation will be marked as an error, and you can drill down into the logs or other telemetry associated with that specific span.
The trace directive in Caddy is surprisingly flexible. Beyond otlp, it supports other exporters like zipkin and jaeger. You can also configure sampling, though Caddy defaults to "always on" for simplicity when tracing is enabled, which is often what you want during initial setup.
The mental model to hold is that Caddy becomes a node in a distributed tracing graph. It doesn’t just forward requests; it actively participates in and annotates the request’s journey with its own operational context.
The most surprising thing is how seamlessly Caddy integrates with the OpenTelemetry ecosystem. It’s not an afterthought; it’s a first-class citizen, and enabling it is as simple as adding a few lines to your Caddyfile, immediately making your Caddy-proxied services observable.
Once you have tracing set up, the next logical step is to correlate these traces with logs and metrics for a complete observability picture.