Argo CD doesn’t expose metrics in a way Prometheus can just pull by default; you have to explicitly tell it to do so and configure Prometheus to pick them up.
Let’s see Argo CD spitting out metrics. Here’s a snippet of what you’d see if you hit the /metrics endpoint on an Argo CD API server that’s been configured for it:
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0.5"} 0.012077615
go_gc_duration_seconds{quantile="0.99"} 0.03257818
go_gc_duration_seconds{quantile="0.999"} 0.03257818
go_gc_duration_seconds sum:go_gc_duration_seconds 0.528844792
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 153
# HELP go_info Go version and build information.
# TYPE go_info gauge
go_info{version="go1.19.10"} 1
# HELP argocd_api_request_duration_seconds API request latencies.
# TYPE argocd_api_request_duration_seconds histogram
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="0.005"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="0.01"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="0.025"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="0.05"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="0.1"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="0.25"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="0.5"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="1"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="2.5"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="5"} 0
argocd_api_request_duration_seconds_bucket{handler="/argocd.ArgoCD/GetApplication",le="+Inf"} 0
argocd_api_request_duration_seconds_sum{handler="/argocd.ArgoCD/GetApplication"} 0
argocd_api_request_duration_seconds_count{handler="/argocd.ArgoCD/GetApplication"} 0
# HELP argocd_redis_connection_status Redis connection status
# TYPE argocd_redis_connection_status gauge
argocd_redis_connection_status 1
This is the Prometheus exposition format. Argo CD, by default, runs an API server and a repo server. You need to enable metrics on at least the API server. The metrics are primarily generated by the Go runtime (like go_goroutines) and then specific Argo CD metrics related to API requests, Git operations, and sync status.
The core problem Argo CD metrics solve is visibility into its operational health and performance. Without them, you’re flying blind on things like:
- API Latency: How long are API calls taking? Are specific endpoints slow?
- Sync Status: How many applications are out of sync? How long do sync operations take?
- Resource Usage: How many goroutines is the API server using? What’s the Go GC activity?
- External Dependencies: Is it connected to Redis? What’s the health of its Git operations?
To get these metrics into Prometheus, you need to:
- Enable Metrics in Argo CD: This is done by setting a command-line flag when you run the Argo CD API server.
- Configure Prometheus to Scrape: Tell your Prometheus instance to look for and collect metrics from the Argo CD API server’s metrics endpoint.
Here’s how you’d typically enable it in a Kubernetes deployment. You’ll modify the argocd-server deployment.
First, find your argocd-server deployment:
kubectl get deployment -n argocd argocd-server -o yaml
You’ll want to add the --metrics flag to the command or args section of the container spec.
Example argocd-server Deployment Snippet (adding the flag):
spec:
template:
spec:
containers:
- command:
- /usr/bin/argocd-server
- --metrics
# ... other args
image: quay.io/argoproj/argocd:v2.9.3 # Use your actual version
name: argocd-server
ports:
- containerPort: 8080 # Default API port
name: http
protocol: TCP
# ... other container spec
If your command is already set, you’ll likely append --metrics to the args list:
spec:
template:
spec:
containers:
- args:
- --metrics # <-- Add this line
- --staticassets=/usr/share/argocd/
# ... other args
command:
- /usr/bin/argocd-server
image: quay.io/argoproj/argocd:v2.9.3
name: argocd-server
# ... other container spec
Apply this change:
kubectl apply -n argocd -f your-modified-argocd-server-deployment.yaml
After applying, the argocd-server pods will restart with the --metrics flag enabled. You can verify by port-forwarding to one of the pods and checking the /metrics endpoint:
# Find a pod name
kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server
# Port forward (replace <pod-name>)
kubectl port-forward -n argocd <pod-name> 8080:8080
# In another terminal, check metrics
curl http://localhost:8080/metrics
You should see the Prometheus-formatted metrics output.
Now, you need to tell Prometheus to scrape these metrics. This is typically done using a ServiceMonitor object if you’re using Prometheus Operator, or by directly configuring prometheus.yml if you’re managing Prometheus manually.
Using Prometheus Operator (ServiceMonitor):
Create a ServiceMonitor object in the same namespace as your Argo CD argocd-server service (usually argocd).
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: argocd-server-metrics
namespace: argocd # Or wherever your Argo CD is installed
labels:
release: prometheus # This label must match your Prometheus instance's selector
spec:
selector:
matchLabels:
app.kubernetes.io/name: argocd-server # This should match the labels on your argocd-server service
namespaceSelector:
matchNames:
- argocd # The namespace where argocd-server runs
endpoints:
- port: http # The name of the port in the Service that exposes metrics
path: /metrics
interval: 30s # How often to scrape
# If your argocd-server service doesn't have a specific port named 'http'
# that exposes metrics, you might need to adjust this.
# The default API port is 8080, so if your service has that,
# this should work.
Apply this ServiceMonitor:
kubectl apply -f your-servicemonitor.yaml
Prometheus Operator will see this ServiceMonitor and automatically configure your Prometheus instance to scrape the argocd-server service’s metrics endpoint.
Manual Prometheus Configuration (prometheus.yml):
If you’re not using Prometheus Operator, you’ll add a scrape job to your prometheus.yml configuration. You’ll need the Kubernetes service name for argocd-server. Assuming it’s argocd-server in the argocd namespace.
scrape_configs:
- job_name: 'argocd-server'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
# Scrape only the argocd-server service endpoints
- source_labels: [__meta_kubernetes_service_name]
action: keep
regex: argocd-server
# Scrape only from the argocd namespace
- source_labels: [__meta_kubernetes_namespace]
action: keep
regex: argocd
# Use the service port (e.g., 8080)
- source_labels: [__meta_kubernetes_endpoint_port_name]
action: keep
regex: http # Or the name of the port exposing metrics
# Set the target URL
- source_labels: [__address__]
regex: (.*)
target_label: __address__
replacement: $1:8080 # Append the port if not already there
# Set the metrics path
- target_label: __metrics_path__
replacement: /metrics
Reload your Prometheus configuration after updating prometheus.yml.
Once Prometheus is scraping, you can explore these metrics in the Prometheus UI (usually at http://<prometheus-host>:9090/graph) or use them in Grafana dashboards. Look for metrics starting with argocd_ to see Argo CD specific data.
The counter-intuitive part about enabling Argo CD metrics is that the --metrics flag alone doesn’t magically expose them on a separate port. It enables the Prometheus exposition format on the existing argocd-server’s default API port (typically 8080). This means your Prometheus scrape configuration needs to target that same port. You’re not setting up a new listener; you’re just telling Prometheus to hit /metrics on the API server’s existing endpoint.
After you have Argo CD metrics in Prometheus, the next logical step is to build Grafana dashboards to visualize them, allowing for easier monitoring and alerting.