CoreDNS is a DNS server that’s become the de facto standard in Kubernetes environments. While it’s incredibly flexible and extensible, its performance and behavior need monitoring, and Prometheus is the go-to tool for that.
Here’s how you can expose and scrape CoreDNS metrics for Prometheus.
First, you need to ensure CoreDNS is configured to expose its metrics. This is typically done via a Prometheus plugin. If you’re running CoreDNS within Kubernetes, this configuration is usually managed by the coredns ConfigMap in the kube-system namespace.
Here’s a snippet of what that ConfigMap might look like, focusing on the Prometheus plugin configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153 # <-- This line is key
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
The crucial part here is prometheus :9153. This directive tells CoreDNS to start an HTTP server on port 9153 and expose Prometheus metrics at the /metrics endpoint. If you’re not using Kubernetes, you’d add this line to your Corefile accordingly.
With the Prometheus plugin enabled, CoreDNS will start listening on port 9153. You can verify this by port-forwarding to the CoreDNS pod (if in Kubernetes) and checking:
kubectl port-forward -n kube-system <coredns-pod-name> 9153:9153
curl http://localhost:9153/metrics
This should return a large output of metrics in Prometheus exposition format, similar to this:
# HELP coredns_build_info CoreDNS build information
# TYPE coredns_build_info gauge
coredns_build_info{version="1.10.1"} 1
# HELP coredns_dns_request_duration_seconds CoreDNS DNS request latency.
# TYPE cdnns_dns_request_duration_seconds histogram
coredns_dns_request_duration_seconds_bucket{le="0.001",type="udp"} 1234
coredns_dns_request_duration_seconds_bucket{le="0.01",type="udp"} 5678
# ... more metrics ...
Now, to scrape these metrics with Prometheus, you need to configure your Prometheus server to discover and scrape the CoreDNS instances. The most common method in Kubernetes is using Prometheus Operator and ServiceMonitors.
First, ensure you have Prometheus Operator installed in your cluster. Then, create a ServiceMonitor resource. This resource tells Prometheus Operator how to find and scrape your services.
Here’s an example ServiceMonitor for CoreDNS:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: coredns
namespace: kube-system # Namespace where CoreDNS is running
labels:
release: prometheus # This label should match your Prometheus release name
spec:
selector:
matchLabels:
k8s-app: coredns # This label should match the labels on your CoreDNS pods/service
namespaceSelector:
matchNames:
- kube-system # Scrape from this namespace
endpoints:
- port: metrics # The name of the port defined in the CoreDNS service
interval: 30s
path: /metrics # The endpoint path where metrics are exposed
targetPort: 9153 # The actual port CoreDNS is listening on for metrics
Make sure the selector.matchLabels in the ServiceMonitor correctly targets your CoreDNS pods or service. Often, CoreDNS pods have the label k8s-app: coredns. The release: prometheus label is crucial for Prometheus Operator to pick up this ServiceMonitor.
The targetPort: 9153 is where Prometheus will actually scrape metrics from. The port: metrics refers to a named port within the Kubernetes Service object that Fronts your CoreDNS pods. If your CoreDNS service doesn’t have a named port metrics pointing to 9153, you might need to adjust your CoreDNS service definition or the ServiceMonitor.
A typical CoreDNS service in Kubernetes looks like this:
apiVersion: v1
kind: Service
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: coredns
spec:
selector:
k8s-app: coredns
ports:
- name: metrics # This name must match the 'port' field in ServiceMonitor
protocol: TCP
port: 9153 # The port the service exposes
targetPort: 9153 # The port on the Pods to send traffic to
Once the ServiceMonitor is applied, Prometheus Operator will detect it. If Prometheus is configured to watch the kube-system namespace (or broadly watches all namespaces), it will automatically add CoreDNS targets to its scrape configuration.
You can then navigate to your Prometheus UI, go to "Status" -> "Targets," and you should see your CoreDNS instances listed with a state of "UP."
The most surprising true thing about this setup is that CoreDNS is effectively running two network servers simultaneously: one for DNS queries (typically UDP/TCP on port 53) and another for metrics (TCP on port 9153). The prometheus :9153 directive in the Corefile simply registers an HTTP handler for /metrics on that port without interfering with the DNS server’s operation.
Once you have the metrics, you can start building dashboards in Grafana. Look for pre-built CoreDNS dashboards or build your own using metrics like coredns_dns_request_duration_seconds_bucket, coredns_dns_requests_total, coredns_network_errors_total, and coredns_cache_entries.
The next concept you’ll likely encounter is using these metrics to set up alerting for DNS resolution failures or performance degradation.