Argo CD’s state is stored in its own Git repository, which is the single source of truth for your desired application state.
Here’s how a typical Argo CD deployment looks:
# Example Argo CD Application manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/myorg/my-app-config.git
targetRevision: HEAD
path: overlays/production
destination:
server: https://kubernetes.default.svc
namespace: my-app
syncPolicy:
automated:
prune: true
selfHeal: true
This manifest tells Argo CD to synchronize the my-app application with the overlays/production directory in the my-app-config.git repository, deploying it to the my-app namespace on the in-cluster Kubernetes API server.
To back up Argo CD for disaster recovery, you need to back up two primary components:
- The Argo CD Application State: This is the metadata Argo CD uses to manage your applications. It includes your
Applicationcustom resources,AppProjectresources, and any other Argo CD-specific configurations. - The Git Repository containing your desired state: This is the source of truth for your application manifests (e.g., deployments, services, ingress).
Backing Up Argo CD Application State
The most robust way to back up Argo CD’s internal state is to back up the Kubernetes Custom Resources that define its configuration. Argo CD stores all its managed application definitions, projects, and other configurations as Kubernetes Custom Resources.
Diagnosis: To see the Argo CD resources, run:
kubectl get application,appproject -n argocd -o yaml
This command lists all Application and AppProject resources in the argocd namespace and outputs them in YAML format. If you have other Argo CD CRDs, you’d list those too.
Common Causes & Fixes:
- Manual
kubectl getandkubectl apply:- Diagnosis: Running
kubectl get application,appproject -n argocd -o yaml > argocd-state-backup.yamlperiodically. - Fix: Automate this process. Use a cron job or a CI/CD pipeline.
- Why it works: This directly captures the current state of your Argo CD managed resources.
- Diagnosis: Running
- Using
argocd-cli:- Diagnosis: The
argocd-clican export application definitions, but it’s more for individual applications and less for a full cluster-wide backup of Argo CD’s own state. - Fix: Focus on backing up the CRDs as shown above. The CLI is for managing applications, not backing up Argo CD’s control plane state.
- Why it works: The CRDs are the definitive representation of Argo CD’s configuration within Kubernetes.
- Diagnosis: The
- Relying solely on GitOps repo:
- Diagnosis: Thinking that your application manifests in Git are enough. This misses Argo CD’s own configuration (like
ApplicationCRs,AppProjectCRs, sync policies, etc.). - Fix: Back up both the GitOps repo and the Argo CD CRDs.
- Why it works: Argo CD’s CRDs define how it interacts with your GitOps repositories and what to deploy, which isn’t fully captured in the application manifests themselves.
- Diagnosis: Thinking that your application manifests in Git are enough. This misses Argo CD’s own configuration (like
- Missing
AppProjectbackups:- Diagnosis: Only backing up
Applicationresources and neglectingAppProject.AppProjectdefines RBAC, source repositories, and destinations, which are critical for security and organization. - Fix: Ensure your backup command includes
appproject:kubectl get appproject -n argocd -o yaml >> argocd-state-backup.yaml. - Why it works:
AppProjectis a distinct CRD that controls access and policy for groups of applications.
- Diagnosis: Only backing up
- Incomplete Backup Scope:
- Diagnosis: Not backing up Argo CD CRDs from the correct namespace. By default, Argo CD resources are in the
argocdnamespace. - Fix: Always specify
-n argocdor ensure your backup tool targets this namespace. If you installed Argo CD elsewhere, adjust the namespace. - Why it works: Kubernetes resources are namespaced; you must target the correct namespace for your backup.
- Diagnosis: Not backing up Argo CD CRDs from the correct namespace. By default, Argo CD resources are in the
- Lack of Version Control for Backups:
- Diagnosis: Storing backup files locally or without versioning. This makes it hard to track changes and roll back to a specific point in time.
- Fix: Commit your YAML backup files to a separate, secure Git repository.
- Why it works: Git provides history, diffs, and a reliable storage mechanism for your backup state.
- No Encryption for Sensitive Data:
- Diagnosis: If your Argo CD CRDs contain secrets (e.g., in
Applicationspec for things likehelm.valuesorkustomize.secretGenerator), these will be backed up in plain text. - Fix: Use Kubernetes secret management solutions (like Sealed Secrets, External Secrets Operator, or Vault) for sensitive data before it gets managed by Argo CD, or ensure your backup repository is highly secured and encrypted.
- Why it works: Prevents sensitive credentials from being exposed in plain text backups.
- Diagnosis: If your Argo CD CRDs contain secrets (e.g., in
Next Steps After Fixing: Once you have a reliable backup of your Argo CD CRDs, your next step is to ensure you can actually restore them. This involves testing the restore process by creating a new Argo CD instance and applying the backed-up YAML.
Backing Up the Git Repository
Your Git repository is the source of truth for your application manifests. This is usually a standard Git repository (GitHub, GitLab, Bitbucket, Gitea, etc.).
Diagnosis: The primary "diagnosis" here is confirming you know which Git repository and which branch/tag Argo CD is configured to use for each application.
Common Causes & Fixes:
- Relying on the Git Provider’s built-in backup:
- Diagnosis: Assuming GitHub/GitLab automatically handles point-in-time recovery for your specific needs. While they have backups, they might not align with your disaster recovery RPO/RTO.
- Fix: Implement your own backup strategy for your Git repositories. This could involve:
- Mirroring: Setting up a read-only mirror of your primary Git repo on a different service or in a different region.
- Regular Dumps: Using
git clone --mirrorandgit remote updatein a cron job to pull the latest state into a separate, archived location. - Git Provider Features: Utilizing features like Gitea’s backup/restore or GitHub Enterprise Server’s backup utilities if available.
- Why it works: Ensures you have an independent copy of your manifests, not subject to the availability or policies of your primary Git hosting provider.
- Not backing up the correct repository/path:
- Diagnosis: Argo CD can be configured with multiple
repoURLs andpaths per application. If you only back up one repository, you might miss others. - Fix: For each
ApplicationCR, identify itsspec.source.repoURLandspec.source.path. Ensure all uniquerepoURLs are backed up. - Why it works: Guarantees that all sources of truth for your deployments are preserved.
- Diagnosis: Argo CD can be configured with multiple
- Forgetting about
targetRevision:- Diagnosis: Backing up the
mainbranch when Argo CD is actually deployed from a specifictagorcommit SHA. - Fix: Your backup strategy should ideally capture all branches and tags, or at least the specific
targetRevisionthat Argo CD is configured to use. If usingHEAD, backing up the current state of the default branch is sufficient. - Why it works: Ensures you can restore to the exact version Argo CD was managing.
- Diagnosis: Backing up the
- Lack of a separate, off-site backup:
- Diagnosis: Storing your Git repository backup in the same cloud provider or data center as your Argo CD instance.
- Fix: Store your Git repository backups in a geographically distinct location or on a different infrastructure provider.
- Why it works: Protects against regional outages or provider-wide failures.
- No integrity checks on backups:
- Diagnosis: Assuming a backed-up Git repository is perfectly intact without verification.
- Fix: Periodically clone your Git backups and run
git fsckto check for integrity issues. - Why it works: Detects corruption in your backup data before you need it.
Next Steps After Fixing: After ensuring your Git repositories are backed up, the next logical step is to understand how to clone a specific revision of your repository and use it to populate a new Argo CD instance’s application definitions.
The Full Disaster Recovery Plan
A complete DR plan for Argo CD involves:
- Automated, Versioned Backups of Argo CD CRDs: Stored securely, preferably in a separate Git repo.
- Automated, Off-site Backups of Application Git Repositories: Ensuring all sources of truth are preserved.
- A Tested Restore Procedure: Documented steps to:
- Provision a new Kubernetes cluster.
- Install Argo CD.
- Restore Argo CD CRDs from your backup.
- Configure Argo CD to point to your backed-up Git repositories (or ensure it can access them).
- Verify applications sync correctly.
The final step in a successful disaster recovery is to ensure that your monitoring and alerting systems are also restored and configured to detect any deviations from your desired state.