EKS control plane logs are actually a stream of audit events, not just isolated log entries.

Here’s how you can enable them and see what’s really going on:

aws eks enable-control-plane-logging \
    --cluster-name my-eks-cluster \
    --log-types api,audit

This command tells EKS to start sending two types of logs to CloudWatch Logs:

  • api: These logs capture all requests made to the Kubernetes API server. Think of it as the raw transcript of every kubectl command, every pod creation, every service update.
  • audit: This is a more detailed, structured subset of the api logs, specifically focusing on security-relevant events. It tells you who did what to which resource, and whether it succeeded or failed.

Once enabled, EKS creates a dedicated log group in CloudWatch Logs for your cluster, typically named /aws/eks/my-eks-cluster/cluster. Within this log group, you’ll find log streams for each control plane component.

Let’s look at what you might see in these logs and why they matter.

Seeing the System in Action: An Audit Event

Imagine you just ran kubectl get pods. This simple command generates an audit event. Here’s a simplified example of what that might look like in your CloudWatch Logs:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "a1b2c3d4-e5f6-7890-1234-abcdef123456",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/pods?limit=500",
  "verb": "get",
  "user": {

    "username": "system:node:{{.Name}}",

    "uid": "...",
    "groups": [
      "system:nodes",
      "system:node-key-adc1234567890abcdef"
    ]
  },
  "sourceIPs": [
    "172.31.1.100"
  ],
  "userAgent": "kubectl/v1.28.0 (linux/amd64) kubernetes/7e3545f",
  "objectRef": {
    "resource": "pods",
    "namespace": "default",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2023-10-27T10:00:00Z",
  "stageTimestamp": "2023-10-27T10:00:01Z",
  "annotations": {
    "authentication.kubernetes.io/podName": "kube-proxy-abcde",
    "authentication.kubernetes.io/podNamespace": "kube-system",
    "authentication.kubernetes.io/containerName": "kube-proxy",
    "authorization.k8s.io/decision": "allow",

    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:node\" of ClusterRole \"system:node\" to (User \"system:node:{{.Name}}\")"

  }
}

Key takeaways from this event:

  • verb: "get": The action performed.
  • objectRef.resource: "pods": The resource targeted.
  • user: Who performed the action. In this case, it’s a node’s kube-proxy, often running kubectl commands to manage services.
  • responseStatus.code: 200: The request was successful.
  • annotations["authorization.k8s.io/decision"]: "allow": The authorization decision. This is crucial for understanding access control.

The Mental Model: Debugging and Security

Enabling control plane logs gives you two primary superpowers:

  1. Deep Debugging: When something goes wrong with your cluster — a pod won’t schedule, a service isn’t accessible, an admission controller is blocking requests — these logs are your first and best source of truth. You can trace the exact API calls that were made, the parameters they used, and the responses received. This is invaluable for pinpointing configuration errors, RBAC issues, or even bugs in Kubernetes itself.
  2. Security Auditing: The audit logs provide a complete history of who did what in your cluster. This is essential for compliance, incident response, and general security posture. You can detect unauthorized access attempts, track changes to critical resources, and understand the blast radius of any security event.

Levers You Control

When you enable logging, EKS handles the backend setup to CloudWatch Logs. The primary "lever" you can pull is the log-types parameter in the enable-control-plane-logging command. The available types are:

  • api: Enables all API server requests. This is the most verbose.
  • audit: Enables security-focused audit events. This is a subset of api logs.
  • authenticator: Logs events related to authentication (e.g., token validation).
  • controllerManager: Logs from the Kubernetes controller manager.
  • scheduler: Logs from the Kubernetes scheduler.

You can specify one or more types. For general visibility and debugging, api and audit are usually the most useful. For deeper troubleshooting of specific Kubernetes components, you might add controllerManager or scheduler.

The One Thing Most People Don’t Know

The level field in the audit event (RequestResponse, Request, Metadata) dictates how much data is logged. The default is RequestResponse, which logs the request body, response body, and metadata. If you’re concerned about log volume or sensitive data exposure, you can configure a custom audit policy. This involves creating a policy file and then telling EKS to use it via the eks update-cluster-config command with the --audit-policy parameter. This allows granular control over what information is captured for different API operations and resources, significantly reducing log noise and improving security.

The next thing you’ll likely encounter is understanding how to filter and query these logs effectively within CloudWatch Logs Insights to extract the specific information you need.

Want structured learning?

Take the full Eks course →