Audit Logs & Investigation (CKS)
Record who did what, then reconstruct an attack from the evidence.
An interactive Kubernetes lesson: 20 steps, about 35 minutes, on a live simulation in your browser.
The shop runs its api in default and a ledger service in a payments namespace. Three identities use the cluster: Dana, a developer with read access; ci-deployer, the ServiceAccount the build pipeline deploys with; and ops-admin, a ServiceAccount with cluster-admin used by an operations tool.
Watch two ordinary requests go through the API server. Dana lists pods. The pipeline lists Secrets. Both pass every gate and are answered.
What you will learn
No record
- Who read that Secret?: Events tell you what the cluster did. Only the audit log tells you who asked it to.
Turning on the audit log
- Four levels of detail: Metadata answers who did what. Request adds what they sent. RequestResponse adds what they got back.
- Break it: flags without files: A static pod sees only what its manifest mounts. A flag that names a host path needs a volume to match.
- Flags, volumes, and the first entry
- Add a rule at the bottom: An audit policy is read like a firewall rule list: top to bottom, first match wins, order is the policy.
- A policy worth keeping: Log bodies for changes, metadata for reads, and only metadata for Secrets. The audit log must not become the leak.
Reading the log
- One line, one request: Every audit event answers the same five questions: who, from where, did what, to which object, with what result.
- A request that was refused: A 403 in the audit log is someone reaching for something they were not given. A burst of them is someone exploring.
- Drill: everything one identity did
Following the trail
- 02:14, a shell in the api: Falco tells you what ran inside a container. The audit log tells you who asked the API for it. An investigation joins the two on pod name and time.
- Initial access: a token from a new address: A known identity from an unknown address, opening with 403s, is a stolen credential being tried out.
- Persistence: a job that comes back: Persistence is anything that restarts the attacker's code without them: a CronJob, a DaemonSet, a new credential.
Escalation and theft
- Privilege escalation: In a namespace, whoever can create pods can act as every ServiceAccount there. The most powerful one sets the ceiling for all.
- Lateral movement: into payments: Attackers change identities. Pivot on what they cannot easily change: source address, time window, user agent.
- Break it: the payment key leaves: Initial access, execution, persistence, privilege escalation, lateral movement, exfiltration: each phase leaves a different kind of line.
Naming the actor, containing it
- Which account was compromised?: The compromised identity is the first one the attacker used, not the most powerful one they reached.
- Contain it: Contain in the order credentials, persistence, footholds, then rotate what was exposed. Anything you skip is a way back in.
- Drill: what can it still do?
Recap & playground
- Cheat sheet
- Playground