Argo CD doesn’t store audit logs by default; it sends them to standard output or a configured webhook, meaning you have to actively set up a destination to capture them.

Let’s see what that looks like in practice. Imagine you have a GitOps workflow where a developer merges a change to your main branch, triggering Argo CD to sync that change to a Kubernetes cluster.

Here’s a snippet of what an audit log entry for that sync operation might look like, assuming you’ve configured a webhook to capture it:

{
  "receivedAt": "2023-10-27T10:30:00Z",
  "agent": {
    "type": "argocd-server",
    "version": "v2.8.4"
  },
  "source": {
    "ip": "192.168.1.100",
    "userAgent": "ArgoCD/v2.8.4"
  },
  "request": {
    "path": "/api/v1/applications/my-app/sync",
    "method": "POST",
    "rpcMethod": "Sync",
    "headers": {
      "Content-Type": "application/json"
    }
  },
  "response": {
    "code": 200,
    "message": "Sync initiated successfully"
  },
  "userInfo": {
    "username": "developer@example.com",
    "groups": ["developers"],
    "iss": "argocd"
  },
  "loggedAt": "2023-10-27T10:30:05Z",
  "operation": {
    "sync": {
      "revision": "a1b2c3d4e5f67890abcdef1234567890abcdef12",
      "resources": [
        {
          "group": "",
          "version": "v1",
          "kind": "Pod",
          "namespace": "default",
          "name": "my-app-deployment-xyz123",
          "status": "Modified"
        }
      ]
    }
  }
}

This log entry tells us:

  • When it happened (receivedAt, loggedAt).
  • Who performed the action (userInfo.username, userInfo.groups).
  • What they did (request.method, request.path, operation.sync).
  • What was affected (operation.sync.resources).
  • What the outcome was (response.code).
  • Where it came from (source.ip).

The primary problem audit logging solves in Argo CD is providing a verifiable trail of who did what and when within your GitOps system. This is crucial for security, compliance, and debugging. Without it, understanding how a change was introduced, who authorized it, or why a particular sync occurred becomes a manual, often impossible, investigation.

To get this data, you need to configure Argo CD. The core of this involves setting the argocd-server’s audit log configuration. You’ll typically do this by modifying the argocd-server’s deployment or by using a ConfigMap if you’re using a Helm chart or a more advanced setup.

1. Enabling Audit Logs to Standard Output (stdout)

This is the simplest way to start. Argo CD will print audit logs to the argocd-server container’s logs.

  • Diagnosis: Check the logs of your argocd-server pod.

    kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server -c argocd-server
    

    If you don’t see JSON entries related to request or userInfo, audit logging isn’t configured or isn’t outputting to stdout.

  • Fix: Add or modify the --audit-log-format and --audit-log-log-level flags in the argocd-server container’s command arguments. Locate your argocd-server deployment (often in the argocd namespace):

    kubectl edit deployment argocd-server -n argocd
    

    Find the containers section for argocd-server and add/ensure these arguments are present under command:

    spec:
      template:
        spec:
          containers:
          - name: argocd-server
            # ... other container settings ...
            command:
            - /usr/bin/argocd-server
            args:
            - --log-format=json
            - --audit-log-format=json
            - --audit-log-log-level=info # or "debug" for more detail
            # ... other args ...
    

    The --audit-log-format=json is key. --audit-log-log-level=info captures standard audit events. debug will include more granular details.

  • Why it works: This tells the argocd-server process to format its audit events as JSON and print them to its standard output stream, which kubectl logs then captures.

2. Enabling Audit Logs to a Webhook

This is where you send logs to an external system (like Splunk, ELK stack, or a custom log collector).

  • Diagnosis: If you’ve configured a webhook but aren’t receiving logs, check your argocd-server logs for any errors related to sending HTTP requests. Also, verify the target webhook endpoint is reachable and accepting POST requests.

  • Fix: Configure the --audit-log-webhook-config flag in the argocd-server container. This flag takes a path to a configuration file. First, create a ConfigMap containing your webhook configuration. For example, create a file named audit-webhook-config.yaml:

    # audit-webhook-config.yaml
    webhooks:
      - name: my-log-collector
        url: https://your-log-collector.example.com/api/v1/audit
        # Optional: Add authentication if your webhook requires it
        # auth:
        #   basicAuth:
        #     username: user
        #     password:
        #       name: log-collector-password-secret
        #       key: password
    

    Then, create the ConfigMap in the argocd namespace:

    kubectl create configmap argocd-audit-webhook-config -n argocd --from-file=audit-webhook-config.yaml
    

    Now, edit your argocd-server deployment:

    kubectl edit deployment argocd-server -n argocd
    

    Add the --audit-log-webhook-config argument to the command section:

    spec:
      template:
        spec:
          containers:
          - name: argocd-server
            # ...
            command:
            - /usr/bin/argocd-server
            args:
            - --log-format=json
            - --audit-log-format=json
            - --audit-log-log-level=info
            - --audit-log-webhook-config=/argocd/config/audit-webhook-config.yaml # Path inside the container
            # ...
          volumes: # Add a volume to mount the configmap
          - name: config-volume
            configMap:
              name: argocd-audit-webhook-config
          # ...
          container:
            # ...
            volumeMounts:
            - name: config-volume
              mountPath: /argocd/config # Directory where the configmap file will be available
    

    Ensure the mountPath in volumeMounts matches the directory specified in the --audit-log-webhook-config argument.

  • Why it works: The ConfigMap makes your webhook configuration accessible within the argocd-server pod. The --audit-log-webhook-config argument tells Argo CD to read this configuration and send audit logs as JSON POST requests to the specified url.

3. Controlling Audit Log Detail Level

You can fine-tune how much information is logged.

  • Diagnosis: If your logs are too verbose or not detailed enough, check the current log level setting.

  • Fix: Adjust the --audit-log-log-level flag.

    • info: Standard audit events (syncs, logins, etc.).
    • debug: More detailed information, including specific resource changes within an operation.
    • warn, error, fatal: For troubleshooting only, will log very little. Modify the args in your argocd-server deployment as shown in the previous examples, changing --audit-log-log-level=info to --audit-log-log-level=debug if you need more detail.
  • Why it works: This flag directly controls the verbosity of the audit logging mechanism within Argo CD, allowing you to balance detail with log volume.

4. Rotating Audit Logs (for stdout)

If you are logging to stdout, Kubernetes itself handles log rotation based on its own configuration (e.g., containerd or docker runtime settings, and Kubernetes node configurations).

  • Diagnosis: If your argocd-server logs are filling up disk space on your nodes, it’s a sign that rotation isn’t happening frequently enough or the log retention period is too long.

  • Fix: This is typically managed at the Kubernetes node level or the container runtime level, not directly by Argo CD. For containerd, you might adjust /etc/containerd/config.toml’s plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.max_container_log_line_size or the max_log_file_size and max_log_file_count under plugins."io.containerd.grpc.v1.cri".containerd.log_directory. For Docker, similar configurations exist in daemon.json. You’d need to restart containerd or docker and potentially redeploy your argocd-server pods for changes to take effect.

  • Why it works: Kubernetes relies on the container runtime to manage container logs. By configuring the runtime’s log rotation policies, you ensure that logs are periodically archived or deleted, preventing disk exhaustion.

5. Handling Authentication for Webhook

If your webhook requires authentication, it needs to be configured securely.

  • Diagnosis: If your webhook configuration is set up but logs aren’t being sent, check argocd-server logs for authentication errors (e.g., 401 Unauthorized).

  • Fix: Use Kubernetes Secrets for sensitive information like passwords. Create a secret:

    kubectl create secret generic log-collector-password-secret -n argocd --from-literal=password='your_super_secret_password'
    

    Then, reference this secret in your audit-webhook-config.yaml as shown in the webhook fix section above. Ensure the argocd-server service account has permissions to read this secret if it’s in a different namespace.

  • Why it works: Storing credentials in a Secret is the Kubernetes-native and secure way to manage sensitive data, preventing them from being exposed in plain text within ConfigMaps or deployment manifests.

Once audit logging is correctly configured, the next thing you’ll likely encounter is how to effectively query and analyze these logs, especially if you’re sending them to a centralized logging system.

Want structured learning?

Take the full Argocd course →