Argo CD’s built-in health checks are surprisingly opinionated, often marking complex applications as "Degraded" or "Unhealthy" when they’re perfectly functional.
Let’s see Argo CD’s default health check in action. Imagine a simple Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app-deployment
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app-container
image: nginx:latest
ports:
- containerPort: 80
When Argo CD syncs this, it defaults to checking Deployment and StatefulSet resources. For a Deployment, it looks for status.readyReplicas to equal spec.replicas. If these match, it’s Healthy. If status.replicas doesn’t match spec.replicas (meaning pods are still starting or failing), it’s Progressing. If status.availableReplicas is 0, it’s Unhealthy.
This works for simple cases. But what about an application that uses a StatefulSet and a Service?
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: my-sts
spec:
replicas: 2
selector:
matchLabels:
app: my-sts
template:
metadata:
labels:
app: my-sts
spec:
containers:
- name: my-app
image: busybox
command: ["sh", "-c", "sleep 3600"]
---
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: my-sts
ports:
- protocol: TCP
port: 80
targetPort: 80
Argo CD will happily mark the StatefulSet as Healthy if readyReplicas == replicas. But what if the Service isn’t actually routing traffic correctly yet? The StatefulSet might be running, but the application isn’t truly ready. This is where custom health checks become essential.
The key to custom health checks lies in the argoproj.io/argocd-custom-health annotation on your Kubernetes resources. You can define a Health object within this annotation, specifying livenessProbe and readinessProbe fields that mirror Kubernetes probe configurations. Argo CD uses these definitions to determine the health status.
Here’s how you’d add a custom health check to our StatefulSet to ensure a file exists within the pod, indicating the application has started its initialization:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: my-sts
annotations:
argoproj.io/argocd-custom-health: |
livenessProbe:
exec:
command: ["cat", "/app/initialized.flag"]
readinessProbe:
exec:
command: ["cat", "/app/initialized.flag"]
spec:
replicas: 2
selector:
matchLabels:
app: my-sts
template:
metadata:
labels:
app: my-sts
spec:
containers:
- name: my-app
image: busybox
command: ["sh", "-c", "echo 'Initializing...' && sleep 10 && echo 'done' > /app/initialized.flag && echo 'Initialized' && sleep 3600"]
In this example, both livenessProbe and readinessProbe use an exec command. Argo CD will execute cat /app/initialized.flag inside each pod. If the file exists and the command returns an exit code of 0, the probe passes. If the file doesn’t exist, the command fails with a non-zero exit code, and Argo CD marks the resource as Unhealthy or Degraded. The command in the container spec is modified to create this file after a short delay.
You can also define custom health checks for other resource types. For example, to check if a Deployment has a specific annotation indicating readiness:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app-deployment
annotations:
argoproj.io/argocd-custom-health: |
livenessProbe:
exec:
command: ["sh", "-c", "echo $MY_VAR"] # Assuming MY_VAR is set in deployment
readinessProbe:
exec:
command: ["sh", "-c", "kubectl get deployment my-app-deployment -o jsonpath='{.metadata.annotations.app.kubernetes.io/ready}' | grep -q 'true'"]
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app-container
image: nginx:latest
ports:
- containerPort: 80
This readinessProbe uses kubectl within the probe’s execution environment to query the deployment itself and check for a specific annotation. This allows for more complex, self-referential health checks.
The argoproj.io/argocd-custom-health annotation supports exec, httpGet, and tcpSocket probes, just like Kubernetes probes. This gives you immense flexibility. You can check for specific HTTP response codes, port availability, or execute arbitrary commands within your pods.
The most surprising aspect of custom health checks is their ability to be applied to any Kubernetes resource type, not just those Argo CD natively tracks for health. You can define custom health for a ConfigMap or even a Namespace if you have a specific readiness condition for them, although this is less common. Argo CD will respect these annotations and use them for its overall application health assessment.
When you combine custom health checks with Argo CD’s sync waves and automated pruning, you can build incredibly robust and self-healing application deployments. The next step is to explore how these custom health checks integrate with Argo CD’s diffing capabilities to identify why a resource is unhealthy, not just that it is.