Lab: Kubernetes cluster security from zero to hero

#Level 0: words and setup

#The words you need

Memory hook

the five locksWho can ask (RBAC) → what can be created (Pod Security) → what can connect (NetworkPolicy) → what protects the data (secrets + encryption) → how strong the sandbox is (RuntimeClass). An attacker only needs one unlocked; you need all five.

#Build the cluster

This reuses the Lima VM from the sandbox lab (it already has Docker). If you don't have it: brew install lima && limactl start --name sandbox-lab, then install Docker inside. Then, inside the VM, install the two tools and create the cluster:

# on your Mac
limactl copy -r kubernetes/labs/cluster-security sandbox-lab:/tmp/k8slab
limactl shell sandbox-lab

# inside the VM, one-time tool install
curl -sSLo kind https://kind.sigs.k8s.io/dl/v0.30.0/kind-linux-arm64 && sudo install kind /usr/local/bin/kind
KV=$(curl -sSL https://dl.k8s.io/release/stable.txt)
curl -sSLo kubectl "https://dl.k8s.io/release/$KV/bin/linux/arm64/kubectl" && sudo install kubectl /usr/local/bin/kubectl

# build the cluster (takes ~20s)
cd /tmp/k8slab && bash setup.sh
export KUBECONFIG=/tmp/kc

setup.sh creates a single-node cluster with API-server audit logging turned on (so you can later inspect who did what, in /var/log/kubernetes/audit.log on the control-plane container). Check it:

$ kubectl get nodes
NAME                   STATUS   ROLES           AGE   VERSION
seclab-control-plane   Ready    control-plane   26s   v1.34.0

Run the experiments in order: bash 01-rbac.sh, then 02, 03, 04, 05. Each is short and commented — read it before running.

#Level 1: RBAC, and the privesc path hiding in "create pods"

Questionif I give a service account exactly "manage pods in one namespace", what did I actually give it?

bash 01-rbac.sh

== Ask the API what deploy-bot can do (kubectl auth can-i --as impersonates it) ==
create pods in team-a?   yes
read secrets in team-a?  no
create pods in kube-system? no

• kubectl auth can-i <verb> <resource> --as=<subject> asks the API server "would you allow this?" without doing it. --as=system:serviceaccount:team-a:deploy-bot impersonates the service account (your admin kubeconfig is allowed to impersonate). This is the single most useful RBAC-debugging command.
• The answers are exactly the Role we bound: pods in team-a, yes; secrets, no; anything in kube-system, no. So far RBAC looks tight.

== The privilege-escalation path hiding in 'create pods' ==
pod/node-escape created

• We just created, as deploy-bot, a pod that mounts the node's entire root filesystem (hostPath: { path: / }) into the container. It was allowed.
• That's the lesson. "Create pods" sounds narrow, but a pod can mount the host, run as root, add capabilities, or use the node's network. deploy-bot can't read secrets through the API, but it can schedule a pod that reads every other pod's secrets off the node's disk, and the kubelet's own credentials with it. RBAC on pods/create quietly includes most of the node. Experiment 2 is the control that actually closes this.

🎯

Exercise. deploy-bot was denied get secrets. Does that mean it can't obtain the database password stored in a Secret? Answer: no. It can create a pod that mounts that Secret as a volume or env var (pods reference secrets by name, and creating the pod doesn't require get secrets), then read it from inside the pod. "Can't read the Secret via the API" and "can't obtain the secret value" are different statements — which is why Pod Security and separate secret stores matter on top of RBAC.

#Level 2: Pod Security turns "any pod" back into "a safe pod"

Questionhow do I stop the host-mounting trick from Level 1 without hand-auditing every Role?

bash 02-pod-security.sh

== Label team-a to enforce the built-in 'restricted' Pod Security standard ==
Warning: existing pods in namespace "team-a" violate the new PodSecurity enforce level "restricted:latest"
Warning: node-escape: allowPrivilegeEscalation != false, unrestricted capabilities, restricted volume types, ...
namespace/team-a labeled

• One label, pod-security.kubernetes.io/enforce=restricted, switches on Pod Security Admission, which is built into the API server (no add-on to install). restricted is the strictest of the three built-in levels (privileged, baseline, restricted).
• It immediately warns that the node-escape pod from Level 1 violates the new level — existing pods aren't killed, but you're told what would now be rejected.

== Retry the exact host-mounting pod from experiment 1, now in the restricted namespace ==
Error from server (Forbidden): ... violates PodSecurity "restricted:latest":
  allowPrivilegeEscalation != false, unrestricted capabilities,
  restricted volume types (volume "host" uses restricted volume type "hostPath"),
  runAsNonRoot != true, seccompProfile ...

• The same pod that succeeded in Level 1 is now Forbidden at admission, before it's ever stored. The error is a checklist of everything unsafe about it — most importantly hostPath, the node-mount trick.
• A pod that follows the rules (drops all capabilities, runs as non-root, no host mounts, seccompProfile: RuntimeDefault) is still accepted. Admission judges the pod's shape, independently of who created it — so it catches the privesc path even when RBAC was too generous.

🎯

Exercise. Your cluster has one namespace that legitimately needs hostPath (a logging agent). How do you keep restricted everywhere else without exceptions weakening the whole cluster? Answer: set the enforce label per namespace — restricted on application namespaces, baseline or privileged only on the specific namespace the agent runs in, ideally where only the platform team has RBAC. Pod Security is per-namespace by design, so the exception is scoped to one blast radius instead of a cluster-wide hole.

#Level 3: NetworkPolicy — from allow-all to default-deny

Questioncan one compromised pod reach every other pod?

bash 03-networkpolicy.sh

== Before any policy: client reaches web (the Kubernetes default is allow-all) ==
<title>Welcome to nginx!</title>

• By default every pod can reach every other pod, cluster-wide. The client pod fetched web's page with no rule permitting it. A flat network: a compromised pod can scan and connect to everything in the namespace (and beyond).

== Apply a default-deny-ingress policy selecting all pods, then retry ==
client → web now:
wget: download timed out

• The policy has an empty podSelector: {} ("every pod in this namespace") and lists Ingress in policyTypes with no allow rules — which means "deny all inbound". The fetch now times out.
• kind's kindnet CNI enforces this. On a cluster whose CNI doesn't (some do nothing with NetworkPolicy), the rule is silently ignored — so if a policy seems to have no effect, check the CNI first. From a default-deny baseline you add back only the flows you want.

🎯

Exercise. You applied default-deny-ingress and web is now unreachable from client — but web can still make outbound calls to the internet and to the metadata service. Why, and how do you stop that? Answer: this policy only covered Ingress (inbound). Egress is still allow-all. Add a second policy with policyTypes: ["Egress"] and explicit allow rules, which is how you stop a compromised pod from exfiltrating data or reaching cloud metadata — the Kubernetes equivalent of the egress firewall in the sandbox lab's Firecracker experiment.

#Level 4: a Secret is base64, not encryption

Questionif I put a password in a Kubernetes Secret, who can read it?

bash 04-secrets.sh

== 1. kubectl shows the value base64-encoded ==
stored as: U3VwM3JTZWNyZXQh
== 2. Decoding is one step for anyone who can read the Secret object ==
decoded:   Sup3rSecret!
== 3. The plaintext sits in etcd itself, with no app involved ==
password
password
Sup3rSecret!

• U3VwM3JTZWNyZXQh is base64, an encoding with no key — base64 -d reverses it instantly. A Secret is not encrypted by being a Secret.
• Step 3 reads the value straight out of etcd, the control-plane database, via the etcd pod: the literal bytes Sup3rSecret! are on disk. Anyone with etcd access — or a stolen etcd backup, or node root — reads every secret in the cluster, no API call needed.
• Two fixes, together: RBAC so few subjects can get secrets (Level 1 denied deploy-bot that), and an EncryptionConfiguration on the API server (aescbc or a KMS provider) so the value is stored as k8s:enc:... and is unreadable at rest. For the crown jewels, an external store (Vault, cloud secrets manager) via CSI so the secret never lands in etcd at all.

🎯

Exercise. You enable encryption at rest today. Is the existing db-cred secret now protected in etcd? Answer: not until it's rewritten. Encryption at rest only applies to writes after you enable it, so you must re-write every existing secret (e.g. kubectl get secrets -A -o json | kubectl replace -f -) to encrypt what's already there. A subtle gap interviewers like: "we turned on encryption" doesn't retroactively protect old data.

#Level 5: RuntimeClass — sending untrusted pods to a stronger sandbox

QuestionPod Security makes a pod well-behaved, but it still shares the node's kernel. How do I run genuinely untrusted code?

bash 05-runtimeclass.sh

== Declare a RuntimeClass named 'gvisor' pointing at the 'runsc' handler ==
runtimeclass.node.k8s.io/gvisor created
== A pod opts in with one line: runtimeClassName: gvisor ==
pod/untrusted created
== Status on this lab cluster (runsc is NOT installed on the kind node) ==
phase: Pending
  Warning  FailedCreatePodSandBox  ... no runtime for "runsc" is configured

• A RuntimeClass just names a handler. A pod opts in with one line, runtimeClassName: gvisor, and the kubelet routes it to that handler instead of the default runc.
• On this lab node gVisor isn't installed, so the pod sits Pending with no runtime for "runsc" is configured. That error is the proof the routing worked: the API server accepted the pod and the kubelet tried to start it under runsc — there was simply no gVisor on the node.
• In production you install gVisor or Kata on a dedicated node pool, taint it so only untrusted work lands there, and require runtimeClassName via policy. What runsc actually does to the pod's syscalls is the whole sandbox-isolation lab — this experiment is the Kubernetes wiring that sends a pod to it.

🎯

Exercise. Why is "Pod Security restricted + NetworkPolicy" not enough for running a customer's arbitrary container, and what does RuntimeClass add? Answer: restricted stops the pod from mounting the host or running as root, and NetworkPolicy limits where it talks — but the container still shares the node's Linux kernel, so one kernel exploit (Dirty Pipe, a bad syscall) escapes to the node regardless of how well-behaved the pod's spec is. RuntimeClass with gVisor/Kata puts a second kernel or a VM between the pod and the host, so a kernel bug hits a disposable sandbox, not the node. The layers compose: well-behaved pod and strong sandbox.

#Clean up

sudo kind delete cluster --name seclab      # removes the cluster and its containers
exit                                        # leave the VM
limactl stop sandbox-lab                    # free the VM's RAM (keeps it for next time)

#If something goes wrong

#Which experiment backs which interview answer