learninfra · Linux · Networking · Kubernetes · System Design · AI Infrastructure · Exam blueprints · Drills

Secrets, Sandboxes & mTLS (CKS)

Encrypt Secrets at rest, sandbox untrusted pods, encrypt traffic between them.

An interactive Kubernetes lesson: 22 steps, about 35 minutes, on a live simulation in your browser.

The shop's payments backend, api, needs a database password. You do the right thing and keep it out of the image and out of Git: it goes into a Secret called db-creds, and the Deployment reads it as an environment variable.

web calls api, api talks to the database, requests come back 200. Look at the Secret in the terminal. The value is not s3cr3t-pg. It is czNjcjN0LXBn.

What you will learn

  1. A Secret is not a secret

    • A password for the api
    • Drill: decode a Secret: base64 is a way to write bytes as text. It protects a Secret exactly as much as writing it in capital letters would.
    • Read it straight out of etcd: By default a Secret is a plain object in etcd. Whoever can read etcd's disk or its backups can read every Secret in the cluster.

  2. Encrypting etcd

    • Turn on encryption at rest: First provider writes, every provider reads. Put the new key first and keep identity last until everything is rewritten.
    • What happened to the old Secret?: Encryption at rest happens on write. Turning it on protects tomorrow's Secrets, not yesterday's.
    • Rewrite every Secret
    • Where does the key live?: Encryption at rest protects copies of etcd. It does not protect against someone who holds both the data and the key.
    • Drill: prove it is encrypted

  3. Who can read it

    • A Role that can only list: On Secrets, get, list and watch are all read access to the values. There is no verb that shows names only.
    • Break it: no access, still reads it: Inside a namespace, permission to create pods is permission to read every Secret in it. Draw namespaces along trust boundaries.
    • Mount it as a file, not as env: Env vars are copied everywhere the process goes. A mounted Secret is one file that the kubelet keeps current.

  4. One kernel for everyone

    • Every container shares one kernel: A container is a fenced-off process, not a machine. All the fences on a node are drawn by the same kernel.
    • A sandbox runtime and a RuntimeClass: RuntimeClass is a name for a runtime handler. The pod asks by name; the node must actually have the handler.
    • The node without the handler
    • Send sandboxed pods to the right nodes: A sandbox gives the container a kernel that is not the node's. Breaking out now takes two escapes, not one.

  5. Tenants side by side

    • Is a namespace a wall?: A namespace is a folder, not a firewall. The isolation comes from what you attach to it.
    • Build the walls: Tenant = namespace + RBAC + quota + Pod Security + default-deny network. Miss one and the wall has a door in it.

  6. Traffic between pods

    • Break it: listen on the wire: NetworkPolicy controls who may talk. It does not hide what is said, and it does not prove who is speaking.
    • Mutual TLS with a service mesh: mTLS is TLS where both ends show a certificate. A mesh gives each workload an identity and does the handshake for it.
    • Encryption in the network layer: Transparent encryption protects the wire between nodes. Mutual authentication proves the workload. You want both.

  7. Recap & playground

    • Cheat sheet
    • Playground