Monitor etcd with Prometheus and Grafana Dashboards (2026)

Etcd doesn’t just store Kubernetes state; it is the Kubernetes state, and if it hiccups, everything else follows.

Here’s how you can get Prometheus sniffing around etcd’s metrics and Grafana visualizing them, so you’re not blindsided when the cluster starts acting like a moody teenager.

Exposing Etcd Metrics

First, etcd needs to be told to serve its metrics. This is usually done via a flag in its static pod manifest. For a typical Kubernetes setup, this manifest lives at /etc/kubernetes/manifests/etcd.yaml on your control plane nodes.

You’ll want to add the --listen-metrics-urls flag to the etcd container’s command arguments.

apiVersion: v1
kind: Pod
metadata:
  name: etcd
  namespace: kube-system
spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://127.0.0.1:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://<YOUR_NODE_IP>:2380
    - --initial-cluster=<YOUR_NODE_NAME>=https://<YOUR_NODE_IP>:2380
    - --initial-cluster-state=existing
    - --name=<YOUR_NODE_NAME>
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --proxy=on
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-metrics-urls=http://127.0.0.1:2381  # Add this line
    image: registry.k8s.io/etcd:3.5.9-0
    ...

Why this works: The --listen-metrics-urls flag tells etcd to expose its internal metrics on a specified HTTP endpoint. By default, Prometheus will scrape this endpoint. http://127.0.0.1:2381 is the common choice, keeping it internal to the node. After adding this, you’ll need to restart the etcd static pod. This typically means kubectl delete pod etcd -n kube-system on each control plane node; Kubernetes will automatically recreate it.

Prometheus Configuration

Now, tell Prometheus to scrape these new metrics. You’ll likely be using a Prometheus Operator or a similar method for managing your Prometheus instance. The key is to add a ServiceMonitor or PodMonitor that targets etcd.

Here’s a sample ServiceMonitor if etcd is running as a static pod and you’ve exposed the metrics port:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd-metrics
  namespace: monitoring # Or wherever your Prometheus is installed
spec:
  selector:
    matchLabels:
      app: etcd # This label needs to match the etcd pod/service
  namespaceSelector:
    matchNames:
    - kube-system # The namespace where etcd is running
  endpoints:
  - port: metrics # This name should match the port name in your Service definition
    interval: 30s
    path: /metrics
    scheme: http
    bearerTokenSecret:
      name: kubeconfig-token
      key: token
    tlsConfig:
      insecureSkipVerify: true # Adjust if you have proper TLS setup for metrics

If you’re not using Prometheus Operator, you’ll manually add a scrape job to your prometheus.yml:

scrape_configs:
  - job_name: 'etcd'
    scheme: http
    static_configs:
      - targets: ['<CONTROL_PLANE_NODE_IP>:2381'] # Replace with your node IPs
    metrics_path: /metrics
    tls_config:
      insecure_skip_verify: true

Why this works: Prometheus uses ServiceMonitor (or PodMonitor) CRDs to discover targets to scrape. This configuration tells Prometheus to look for pods with the label app: etcd in the kube-system namespace and scrape the metrics port on them. If you’re manually configuring, you’re directly telling Prometheus which IP addresses and ports to poll for metrics. The bearerTokenSecret is often needed for Prometheus to authenticate with the Kubernetes API to discover targets, especially if using ServiceMonitor.

Grafana Dashboards

Once Prometheus is collecting etcd metrics, you can import pre-built Grafana dashboards. The most popular one is often referred to as "Etcd Cluster" or similar. You can find these on Grafana’s official dashboard repository or other community sites.

1. Go to your Grafana instance.
2. Navigate to "Dashboards" -> "Import".
3. Enter a dashboard ID (e.g., 11660 for a common etcd cluster dashboard) or upload a JSON file.
4. Select your Prometheus data source.
5. Click "Import".

The dashboard will populate with key etcd metrics like:

• Leader Changes: Frequent leader changes indicate instability.
• WAL Sync Duration: High WAL sync times mean etcd is slow to durably write data, impacting performance.
• Proposal Fail Rate: Indicates issues with etcd nodes being unable to agree on state.
• Disk I/O: Etcd is heavily disk-bound; slow disks directly translate to slow etcd.
• Network Latency: High latency between etcd peers causes timeouts and instability.

Why this works: These dashboards are pre-configured with PromQL queries designed to extract meaningful insights from the raw etcd metrics. They provide a visual representation of etcd’s health and performance, making it easy to spot anomalies and diagnose problems.

Key Metrics to Watch

• etcd_server_leader_changes_seen_total: Tracks how often the etcd leader has changed. A high rate is a red flag for instability.
• etcd_wal_fsync_duration_seconds_bucket: Shows how long it takes etcd to persist its Write-Ahead Log (WAL) to disk. High values (especially percentiles like 99%) point to disk I/O bottlenecks.
• etcd_network_peer_round_trip_time_seconds_bucket: Measures the latency between etcd peers. High latency degrades cluster performance and can lead to leader elections.
• etcd_server_proposals_failed_total: Counts failed proposals, which are attempts by etcd to commit a transaction. A rising count suggests consensus issues.
• process_resident_memory_bytes: Standard memory usage for the etcd process. Watch for leaks or excessive consumption.

The next thing you’ll likely encounter is Prometheus reporting etcd targets as DOWN if the listen-metrics-urls flag was mistyped or if network policies are blocking Prometheus from reaching the metrics endpoint.