Kubernetes Audit Logging

Required knowledge for the CKS certification.

Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The audit system captures activities generated by users, applications using the Kubernetes API, and the control plane itself.

Audit records allow operators to answer key security questions: what happened, when it happened, who initiated the request, on what resource it was performed, where the event was observed, and from where the request originated.

---

Audit Stages

Each request generates an audit event at one of the following stages:

• RequestReceived — The event is generated as soon as the audit handler receives the request, before it is processed.
• ResponseStarted — The event is generated once the response headers are sent, before the response body is transmitted. This stage applies only to long-running requests, such as watch.
• ResponseComplete — The event is generated after the response body is fully transmitted; no more bytes will be sent.
• Panic — The event is generated when a panic occurs during request processing.

---

Audit Levels

When a request event is evaluated against the audit policy, the first matching rule determines the audit level applied. The available levels are:

• None — Do not log events matching this rule.
• Metadata — Log event metadata (requesting user, timestamp, resource, verb) but not the request or response body.
• Request — Log event metadata and the request body, but not the response body. This level does not apply to non-resource requests.
• RequestResponse — Log event metadata, the request body, and the response body. This level does not apply to non-resource requests.

---

Configuring an Audit Policy

Pass a policy file to kube-apiserver using the --audit-policy-file flag. The rules field must be provided; a policy with zero rules is treated as invalid.

The first matching rule in the policy determines the audit level applied to the event. Use specific rules early in the list to avoid unintended catch-all behavior.

Minimal Policy

The following policy logs all requests at the Metadata level:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

Production Policy Example

The following policy demonstrates common security-oriented patterns:

apiVersion: audit.k8s.io/v1
kind: Policy
# Do not generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["pods"]

  # Log pod/log and pods/status at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Do not log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Do not log watch requests by "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: ""
      resources: ["endpoints", "services"]

  # Do not log authenticated requests to certain non-resource URL paths
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*"
    - "/version"

  # Log the request body of configmap changes in kube-system
  - level: Request
    resources:
    - group: ""
      resources: ["configmaps"]
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at Request level
  - level: Request
    resources:
    - group: ""
    - group: "extensions"

  # Catch-all rule to log all other requests at Metadata level
  - level: Metadata
    omitStages:
      - "RequestReceived"

---

Enabling the Log Backend

The log backend writes audit events to a file in JSONlines format. Configure it with the following kube-apiserver flags:

• --audit-log-path — Path to the log file. Use - to write to stdout. This flag is required to enable the log backend.
• --audit-log-maxage — Maximum number of days to retain old audit log files.
• --audit-log-maxbackup — Maximum number of audit log files to retain.
• --audit-log-maxsize — Maximum size in megabytes before the log file is rotated.

Add both --audit-policy-file and --audit-log-path to the kube-apiserver manifest:

- --audit-policy-file=/etc/kubernetes/audit-policy.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
- --audit-log-maxage=30
- --audit-log-maxbackup=10
- --audit-log-maxsize=100

Volume Mounts for Static Pod Control Planes

When the API server runs as a static pod (as in clusters provisioned with kubeadm), mount the policy file and log directory as host volumes:

volumeMounts:
  - mountPath: /etc/kubernetes/audit-policy.yaml
    name: audit
    readOnly: true
  - mountPath: /var/log/kubernetes/audit/
    name: audit-log
    readOnly: false

volumes:
- name: audit
  hostPath:
    path: /etc/kubernetes/audit-policy.yaml
    type: File
- name: audit-log
  hostPath:
    path: /var/log/kubernetes/audit/
    type: DirectoryOrCreate

---

Enabling the Webhook Backend

The webhook backend sends audit events to a remote endpoint that implements the Kubernetes API conventions. Configure it with the following flags:

• --audit-webhook-config-file — Path to a file with the webhook configuration in a specialized kubeconfig format.
• --audit-webhook-initial-backoff — Initial backoff duration before retrying after a failed webhook request.

---

Considerations

Issue: Audit logging increases memory consumption of the API server, as context required for each request must be stored for the duration of processing.
Fix: Use targeted audit policies that limit high-verbosity levels (Request, RequestResponse) to specific resources (such as secrets and pods) rather than applying them globally.

Issue: Storing audit logs on the same node as the API server allows an attacker who compromises the node to tamper with or delete log files.
Fix: Use the webhook backend to forward events to a remote, append-only log aggregation system in addition to or instead of local log files.

Issue: A policy with omitStages: ["RequestReceived"] reduces log volume but may miss events that were received but not completed (for example, requests dropped under load).
Fix: Apply omitStages selectively to catch-all rules, and preserve RequestReceived logging for high-value resources such as secrets and authentication-related paths.

---

References

This article is based on information from the following official sources:

1. Auditing - Kubernetes Documentation