Orphaned resources in Argo CD are Kubernetes objects that Argo CD believes it’s managing, but they don’t actually exist in the cluster.
Common Causes and Fixes for Orphaned Resources
This usually happens when resources are manually deleted from the cluster outside of Argo CD’s control, or when Argo CD’s state becomes desynchronized with the actual cluster state.
-
Manual Deletion of Resources:
- Diagnosis: Check the
argocd-repo-serverlogs for messages indicating a resource was not found during sync. You can also usekubectl get <resource-type> <resource-name> -n <namespace>to see if the resource is actually present in the cluster. Ifkubectlreturns "not found," and Argo CD still lists it, it’s orphaned. - Fix: If the resource was intentionally deleted and should no longer be managed by Argo CD, you need to prune it from Argo CD’s state. Use the Argo CD CLI:
argocd app resource rm <application-name> --resource <resource-type>:<resource-name> --namespace <resource-namespace>. This command tells Argo CD to stop tracking the resource. - Why it works: This command directly updates Argo CD’s internal state database, removing the reference to the non-existent resource.
- Diagnosis: Check the
-
Failed Deletion or Cleanup:
- Diagnosis: Sometimes, a resource deletion might fail partially, leaving the resource "gone" from the cluster’s perspective but its finalizers still present, or leaving orphaned child resources. Argo CD might still see the parent resource as existing due to a stale watch or a partial sync. Check
kubectl get <resource-type> <resource-name> -n <namespace> -o yamlfor finalizers, orkubectl get events -n <namespace>for related errors. - Fix: If finalizers are the issue, you might need to manually remove them (with caution):
kubectl patch <resource-type> <resource-name> -n <namespace> -p '{"metadata": {"finalizers": []}}'. After manual cleanup, refresh Argo CD’s state:argocd app sync <application-name>. - Why it works: Removing finalizers allows Kubernetes to fully garbage collect the resource.
argocd app syncthen forces Argo CD to re-evaluate the state and realize the resource is gone.
- Diagnosis: Sometimes, a resource deletion might fail partially, leaving the resource "gone" from the cluster’s perspective but its finalizers still present, or leaving orphaned child resources. Argo CD might still see the parent resource as existing due to a stale watch or a partial sync. Check
-
Argo CD Sync Issues or Bugs:
- Diagnosis: If Argo CD syncs fail repeatedly without clear external causes, or if you’re on an older version, a bug might be causing state desynchronization. Examine
argocd-repo-serverandargocd-application-controllerlogs for specific errors during sync operations. - Fix: Ensure you are running the latest stable version of Argo CD. If a bug is suspected, consult the Argo CD GitHub issues. A temporary workaround might involve manually forcing a reconciliation:
argocd app refresh <application-name> --force. - Why it works:
argocd app refresh --forceinstructs the application controller to perform a deep reconciliation, ignoring cached states and re-querying the cluster and Git repository from scratch.
- Diagnosis: If Argo CD syncs fail repeatedly without clear external causes, or if you’re on an older version, a bug might be causing state desynchronization. Examine
-
Resource Managed by Multiple Controllers:
- Diagnosis: If a resource is being managed by both Argo CD and another controller (e.g., a cluster-level operator, Helm controller, or a different GitOps tool), conflicts can arise. Argo CD might try to update or delete a resource that the other controller immediately recreates or modifies. Check the
metadata.ownerReferencesfield of the resource in the cluster. - Fix: Decide on a single source of truth. If Argo CD should manage it, remove it from the other controller’s management. If another controller should manage it, remove the resource from Argo CD’s application definition and prune it from Argo CD’s state using
argocd app resource rm. - Why it works: Eliminating conflicting management ensures only one system is responsible for the resource’s lifecycle, preventing desynchronization.
- Diagnosis: If a resource is being managed by both Argo CD and another controller (e.g., a cluster-level operator, Helm controller, or a different GitOps tool), conflicts can arise. Argo CD might try to update or delete a resource that the other controller immediately recreates or modifies. Check the
-
Cluster State Drift / External Changes:
- Diagnosis: Resources might have been modified in ways that Argo CD doesn’t expect. This is not strictly an "orphaned" resource issue but can lead to Argo CD reporting resources as missing if it expects specific configurations. Use
argocd app diff <application-name>to see what Argo CD thinks is different. If the diff shows resources as missing that you know exist, it points to a state mismatch. - Fix: If the external changes are intentional, update your Git repository to match the cluster state. If the changes are unintentional, revert them in the cluster and let Argo CD reconcile. If you need to "heal" the cluster state from Git:
argocd app sync <application-name>. - Why it works: This process aligns the Git definition with the actual cluster state, resolving discrepancies.
- Diagnosis: Resources might have been modified in ways that Argo CD doesn’t expect. This is not strictly an "orphaned" resource issue but can lead to Argo CD reporting resources as missing if it expects specific configurations. Use
-
Argo CD Internal Database Corruption (Rare):
- Diagnosis: In very rare cases, the etcd database backing Argo CD can become inconsistent. This might manifest as resources appearing orphaned that are definitely present, or vice-versa, without clear logs. This is hard to diagnose definitively without deep inspection of Argo CD’s state.
- Fix: A common approach is to restart Argo CD components, particularly
argocd-repo-serverandargocd-application-controller. If that doesn’t work, a more drastic step might be to re-seed the Argo CD database from Git, which involves deleting and re-importing applications. - Why it works: Restarts can clear transient in-memory inconsistencies. Re-seeding forces Argo CD to rebuild its state entirely from the source of truth (Git).
After fixing orphaned resources, you might encounter Out of sync status for other resources if their desired state in Git doesn’t match the cluster, or Degraded status if a resource’s health condition is not met.