Argo CD doesn’t store audit logs by default; it sends them to standard output or a configured webhook, meaning you have to actively set up a destination to capture them.
Let’s see what that looks like in practice. Imagine you have a GitOps workflow where a developer merges a change to your main branch, triggering Argo CD to sync that change to a Kubernetes cluster.
Here’s a snippet of what an audit log entry for that sync operation might look like, assuming you’ve configured a webhook to capture it:
{
"receivedAt": "2023-10-27T10:30:00Z",
"agent": {
"type": "argocd-server",
"version": "v2.8.4"
},
"source": {
"ip": "192.168.1.100",
"userAgent": "ArgoCD/v2.8.4"
},
"request": {
"path": "/api/v1/applications/my-app/sync",
"method": "POST",
"rpcMethod": "Sync",
"headers": {
"Content-Type": "application/json"
}
},
"response": {
"code": 200,
"message": "Sync initiated successfully"
},
"userInfo": {
"username": "developer@example.com",
"groups": ["developers"],
"iss": "argocd"
},
"loggedAt": "2023-10-27T10:30:05Z",
"operation": {
"sync": {
"revision": "a1b2c3d4e5f67890abcdef1234567890abcdef12",
"resources": [
{
"group": "",
"version": "v1",
"kind": "Pod",
"namespace": "default",
"name": "my-app-deployment-xyz123",
"status": "Modified"
}
]
}
}
}
This log entry tells us:
- When it happened (
receivedAt,loggedAt). - Who performed the action (
userInfo.username,userInfo.groups). - What they did (
request.method,request.path,operation.sync). - What was affected (
operation.sync.resources). - What the outcome was (
response.code). - Where it came from (
source.ip).
The primary problem audit logging solves in Argo CD is providing a verifiable trail of who did what and when within your GitOps system. This is crucial for security, compliance, and debugging. Without it, understanding how a change was introduced, who authorized it, or why a particular sync occurred becomes a manual, often impossible, investigation.
To get this data, you need to configure Argo CD. The core of this involves setting the argocd-server’s audit log configuration. You’ll typically do this by modifying the argocd-server’s deployment or by using a ConfigMap if you’re using a Helm chart or a more advanced setup.
1. Enabling Audit Logs to Standard Output (stdout)
This is the simplest way to start. Argo CD will print audit logs to the argocd-server container’s logs.
-
Diagnosis: Check the logs of your
argocd-serverpod.kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server -c argocd-serverIf you don’t see JSON entries related to
requestoruserInfo, audit logging isn’t configured or isn’t outputting to stdout. -
Fix: Add or modify the
--audit-log-formatand--audit-log-log-levelflags in theargocd-servercontainer’s command arguments. Locate yourargocd-serverdeployment (often in theargocdnamespace):kubectl edit deployment argocd-server -n argocdFind the
containerssection forargocd-serverand add/ensure these arguments are present undercommand:spec: template: spec: containers: - name: argocd-server # ... other container settings ... command: - /usr/bin/argocd-server args: - --log-format=json - --audit-log-format=json - --audit-log-log-level=info # or "debug" for more detail # ... other args ...The
--audit-log-format=jsonis key.--audit-log-log-level=infocaptures standard audit events.debugwill include more granular details. -
Why it works: This tells the
argocd-serverprocess to format its audit events as JSON and print them to its standard output stream, whichkubectl logsthen captures.
2. Enabling Audit Logs to a Webhook
This is where you send logs to an external system (like Splunk, ELK stack, or a custom log collector).
-
Diagnosis: If you’ve configured a webhook but aren’t receiving logs, check your
argocd-serverlogs for any errors related to sending HTTP requests. Also, verify the target webhook endpoint is reachable and accepting POST requests. -
Fix: Configure the
--audit-log-webhook-configflag in theargocd-servercontainer. This flag takes a path to a configuration file. First, create aConfigMapcontaining your webhook configuration. For example, create a file namedaudit-webhook-config.yaml:# audit-webhook-config.yaml webhooks: - name: my-log-collector url: https://your-log-collector.example.com/api/v1/audit # Optional: Add authentication if your webhook requires it # auth: # basicAuth: # username: user # password: # name: log-collector-password-secret # key: passwordThen, create the
ConfigMapin theargocdnamespace:kubectl create configmap argocd-audit-webhook-config -n argocd --from-file=audit-webhook-config.yamlNow, edit your
argocd-serverdeployment:kubectl edit deployment argocd-server -n argocdAdd the
--audit-log-webhook-configargument to thecommandsection:spec: template: spec: containers: - name: argocd-server # ... command: - /usr/bin/argocd-server args: - --log-format=json - --audit-log-format=json - --audit-log-log-level=info - --audit-log-webhook-config=/argocd/config/audit-webhook-config.yaml # Path inside the container # ... volumes: # Add a volume to mount the configmap - name: config-volume configMap: name: argocd-audit-webhook-config # ... container: # ... volumeMounts: - name: config-volume mountPath: /argocd/config # Directory where the configmap file will be availableEnsure the
mountPathinvolumeMountsmatches the directory specified in the--audit-log-webhook-configargument. -
Why it works: The
ConfigMapmakes your webhook configuration accessible within theargocd-serverpod. The--audit-log-webhook-configargument tells Argo CD to read this configuration and send audit logs as JSON POST requests to the specifiedurl.
3. Controlling Audit Log Detail Level
You can fine-tune how much information is logged.
-
Diagnosis: If your logs are too verbose or not detailed enough, check the current log level setting.
-
Fix: Adjust the
--audit-log-log-levelflag.info: Standard audit events (syncs, logins, etc.).debug: More detailed information, including specific resource changes within an operation.warn,error,fatal: For troubleshooting only, will log very little. Modify theargsin yourargocd-serverdeployment as shown in the previous examples, changing--audit-log-log-level=infoto--audit-log-log-level=debugif you need more detail.
-
Why it works: This flag directly controls the verbosity of the audit logging mechanism within Argo CD, allowing you to balance detail with log volume.
4. Rotating Audit Logs (for stdout)
If you are logging to stdout, Kubernetes itself handles log rotation based on its own configuration (e.g., containerd or docker runtime settings, and Kubernetes node configurations).
-
Diagnosis: If your
argocd-serverlogs are filling up disk space on your nodes, it’s a sign that rotation isn’t happening frequently enough or the log retention period is too long. -
Fix: This is typically managed at the Kubernetes node level or the container runtime level, not directly by Argo CD. For
containerd, you might adjust/etc/containerd/config.toml’splugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.max_container_log_line_sizeor themax_log_file_sizeandmax_log_file_countunderplugins."io.containerd.grpc.v1.cri".containerd.log_directory. For Docker, similar configurations exist indaemon.json. You’d need to restartcontainerdordockerand potentially redeploy yourargocd-serverpods for changes to take effect. -
Why it works: Kubernetes relies on the container runtime to manage container logs. By configuring the runtime’s log rotation policies, you ensure that logs are periodically archived or deleted, preventing disk exhaustion.
5. Handling Authentication for Webhook
If your webhook requires authentication, it needs to be configured securely.
-
Diagnosis: If your webhook configuration is set up but logs aren’t being sent, check
argocd-serverlogs for authentication errors (e.g., 401 Unauthorized). -
Fix: Use Kubernetes
Secretsfor sensitive information like passwords. Create a secret:kubectl create secret generic log-collector-password-secret -n argocd --from-literal=password='your_super_secret_password'Then, reference this secret in your
audit-webhook-config.yamlas shown in the webhook fix section above. Ensure theargocd-serverservice account has permissions to read this secret if it’s in a different namespace. -
Why it works: Storing credentials in a
Secretis the Kubernetes-native and secure way to manage sensitive data, preventing them from being exposed in plain text withinConfigMapsor deployment manifests.
Once audit logging is correctly configured, the next thing you’ll likely encounter is how to effectively query and analyze these logs, especially if you’re sending them to a centralized logging system.