Kubernetes Audit Telemetry for Detection Engineers

The audit log observes the API server, not the whole cluster

Kubernetes auditing records security-relevant API activity generated by users, applications, and control-plane components. Each record helps answer who requested what operation on which resource, when, and how the API server handled it.

That vantage point has boundaries. A container process can act without calling the API server, and a node-level event may require runtime or host telemetry. An accepted pod specification does not prove every container later executed exactly as described.

Begin a hypothesis by asking whether the behavior necessarily crosses the API server. If it does not, audit data can provide context but cannot be the sole observation source.

Stage tells you when the event was observed

Kubernetes defines audit stages including RequestReceived, ResponseStarted, ResponseComplete, and Panic. RequestReceived occurs before the handler chain; ResponseComplete occurs after the response body is complete. ResponseStarted is used for long-running requests such as watches.

Do not count stages as independent operations. One request can create more than one event with the same audit identifier. Choose the stage whose fields and outcome match the question, then deduplicate or correlate explicitly.

A received request establishes an attempt, while a completed response can support an outcome assessment. Neither alone proves later workload effect.

Audit level determines which evidence exists

An audit policy selects the first matching rule and assigns a level. Metadata records request metadata without request or response bodies. Request adds the request body, while RequestResponse can add both bodies. None omits matching events.

More detail is not automatically better. Bodies can contain secrets or personal data and increase API-server memory and storage cost. Choose level from the fields a security decision genuinely needs, then protect access and retention.

Record policy version and rule order with the data contract. A rule inserted earlier can silently change which level applies to later traffic.

Identity and authorization require careful roles

The event can include authenticated user information, impersonation, source addresses, verb, object reference, response status, and annotations. Distinguish the authenticated identity, impersonated identity, service account, actor, and target resource before attributing behavior.

Authorization metadata can explain why a request was allowed, but successful authorization does not make behavior benign. Compromised credentials can exercise legitimate permissions. Conversely, denied attempts can be important even though no object changed.

Normalize Kubernetes verbs and resource scope without erasing subresources, namespaces, API groups, or tenant context. Those details often separate routine controllers from risky human or workload activity.

Detection quality depends on controllers, volume, and health

Controllers continually reconcile desired state and can generate high-volume legitimate activity. Baseline by controller identity, resource, namespace, operation, and ownership rather than declaring rare verbs suspicious everywhere. Watch for changes in those relationships.

Health monitoring should cover enabled policy, expected control events, backend delivery, partitions, lag, parse quality, stage distribution, and unknown values. Silence is ambiguous when policy can omit events.

Join audit evidence with admission, workload, identity, and cloud control-plane detection only when entity and time semantics are explicit. The resulting assessment should state which cluster behavior remains outside the audit vantage point.

Imagine a deployment created by a familiar controller after its role binding changed. The creation event is ordinary in isolation; the changed authority alters its meaning. Preserve both facts and their order. That lets you explain a behavioral chain without pretending the audit log observed the controller’s motive or every action inside the resulting workload.