Argo CD’s state is stored in its own Git repository, which is the single source of truth for your desired application state.

Here’s how a typical Argo CD deployment looks:

# Example Argo CD Application manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/my-app-config.git
    targetRevision: HEAD
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

This manifest tells Argo CD to synchronize the my-app application with the overlays/production directory in the my-app-config.git repository, deploying it to the my-app namespace on the in-cluster Kubernetes API server.

To back up Argo CD for disaster recovery, you need to back up two primary components:

  1. The Argo CD Application State: This is the metadata Argo CD uses to manage your applications. It includes your Application custom resources, AppProject resources, and any other Argo CD-specific configurations.
  2. The Git Repository containing your desired state: This is the source of truth for your application manifests (e.g., deployments, services, ingress).

Backing Up Argo CD Application State

The most robust way to back up Argo CD’s internal state is to back up the Kubernetes Custom Resources that define its configuration. Argo CD stores all its managed application definitions, projects, and other configurations as Kubernetes Custom Resources.

Diagnosis: To see the Argo CD resources, run:

kubectl get application,appproject -n argocd -o yaml

This command lists all Application and AppProject resources in the argocd namespace and outputs them in YAML format. If you have other Argo CD CRDs, you’d list those too.

Common Causes & Fixes:

  1. Manual kubectl get and kubectl apply:
    • Diagnosis: Running kubectl get application,appproject -n argocd -o yaml > argocd-state-backup.yaml periodically.
    • Fix: Automate this process. Use a cron job or a CI/CD pipeline.
    • Why it works: This directly captures the current state of your Argo CD managed resources.
  2. Using argocd-cli:
    • Diagnosis: The argocd-cli can export application definitions, but it’s more for individual applications and less for a full cluster-wide backup of Argo CD’s own state.
    • Fix: Focus on backing up the CRDs as shown above. The CLI is for managing applications, not backing up Argo CD’s control plane state.
    • Why it works: The CRDs are the definitive representation of Argo CD’s configuration within Kubernetes.
  3. Relying solely on GitOps repo:
    • Diagnosis: Thinking that your application manifests in Git are enough. This misses Argo CD’s own configuration (like Application CRs, AppProject CRs, sync policies, etc.).
    • Fix: Back up both the GitOps repo and the Argo CD CRDs.
    • Why it works: Argo CD’s CRDs define how it interacts with your GitOps repositories and what to deploy, which isn’t fully captured in the application manifests themselves.
  4. Missing AppProject backups:
    • Diagnosis: Only backing up Application resources and neglecting AppProject. AppProject defines RBAC, source repositories, and destinations, which are critical for security and organization.
    • Fix: Ensure your backup command includes appproject: kubectl get appproject -n argocd -o yaml >> argocd-state-backup.yaml.
    • Why it works: AppProject is a distinct CRD that controls access and policy for groups of applications.
  5. Incomplete Backup Scope:
    • Diagnosis: Not backing up Argo CD CRDs from the correct namespace. By default, Argo CD resources are in the argocd namespace.
    • Fix: Always specify -n argocd or ensure your backup tool targets this namespace. If you installed Argo CD elsewhere, adjust the namespace.
    • Why it works: Kubernetes resources are namespaced; you must target the correct namespace for your backup.
  6. Lack of Version Control for Backups:
    • Diagnosis: Storing backup files locally or without versioning. This makes it hard to track changes and roll back to a specific point in time.
    • Fix: Commit your YAML backup files to a separate, secure Git repository.
    • Why it works: Git provides history, diffs, and a reliable storage mechanism for your backup state.
  7. No Encryption for Sensitive Data:
    • Diagnosis: If your Argo CD CRDs contain secrets (e.g., in Application spec for things like helm.values or kustomize.secretGenerator), these will be backed up in plain text.
    • Fix: Use Kubernetes secret management solutions (like Sealed Secrets, External Secrets Operator, or Vault) for sensitive data before it gets managed by Argo CD, or ensure your backup repository is highly secured and encrypted.
    • Why it works: Prevents sensitive credentials from being exposed in plain text backups.

Next Steps After Fixing: Once you have a reliable backup of your Argo CD CRDs, your next step is to ensure you can actually restore them. This involves testing the restore process by creating a new Argo CD instance and applying the backed-up YAML.

Backing Up the Git Repository

Your Git repository is the source of truth for your application manifests. This is usually a standard Git repository (GitHub, GitLab, Bitbucket, Gitea, etc.).

Diagnosis: The primary "diagnosis" here is confirming you know which Git repository and which branch/tag Argo CD is configured to use for each application.

Common Causes & Fixes:

  1. Relying on the Git Provider’s built-in backup:
    • Diagnosis: Assuming GitHub/GitLab automatically handles point-in-time recovery for your specific needs. While they have backups, they might not align with your disaster recovery RPO/RTO.
    • Fix: Implement your own backup strategy for your Git repositories. This could involve:
      • Mirroring: Setting up a read-only mirror of your primary Git repo on a different service or in a different region.
      • Regular Dumps: Using git clone --mirror and git remote update in a cron job to pull the latest state into a separate, archived location.
      • Git Provider Features: Utilizing features like Gitea’s backup/restore or GitHub Enterprise Server’s backup utilities if available.
    • Why it works: Ensures you have an independent copy of your manifests, not subject to the availability or policies of your primary Git hosting provider.
  2. Not backing up the correct repository/path:
    • Diagnosis: Argo CD can be configured with multiple repoURLs and paths per application. If you only back up one repository, you might miss others.
    • Fix: For each Application CR, identify its spec.source.repoURL and spec.source.path. Ensure all unique repoURLs are backed up.
    • Why it works: Guarantees that all sources of truth for your deployments are preserved.
  3. Forgetting about targetRevision:
    • Diagnosis: Backing up the main branch when Argo CD is actually deployed from a specific tag or commit SHA.
    • Fix: Your backup strategy should ideally capture all branches and tags, or at least the specific targetRevision that Argo CD is configured to use. If using HEAD, backing up the current state of the default branch is sufficient.
    • Why it works: Ensures you can restore to the exact version Argo CD was managing.
  4. Lack of a separate, off-site backup:
    • Diagnosis: Storing your Git repository backup in the same cloud provider or data center as your Argo CD instance.
    • Fix: Store your Git repository backups in a geographically distinct location or on a different infrastructure provider.
    • Why it works: Protects against regional outages or provider-wide failures.
  5. No integrity checks on backups:
    • Diagnosis: Assuming a backed-up Git repository is perfectly intact without verification.
    • Fix: Periodically clone your Git backups and run git fsck to check for integrity issues.
    • Why it works: Detects corruption in your backup data before you need it.

Next Steps After Fixing: After ensuring your Git repositories are backed up, the next logical step is to understand how to clone a specific revision of your repository and use it to populate a new Argo CD instance’s application definitions.

The Full Disaster Recovery Plan

A complete DR plan for Argo CD involves:

  1. Automated, Versioned Backups of Argo CD CRDs: Stored securely, preferably in a separate Git repo.
  2. Automated, Off-site Backups of Application Git Repositories: Ensuring all sources of truth are preserved.
  3. A Tested Restore Procedure: Documented steps to:
    • Provision a new Kubernetes cluster.
    • Install Argo CD.
    • Restore Argo CD CRDs from your backup.
    • Configure Argo CD to point to your backed-up Git repositories (or ensure it can access them).
    • Verify applications sync correctly.

The final step in a successful disaster recovery is to ensure that your monitoring and alerting systems are also restored and configured to detect any deviations from your desired state.

Want structured learning?

Take the full Argocd course →