Kubernetes Incident Response

Production containers run one defined process and shouldn't have interactive shells, so I treat it as a likely compromise. I scope first without tipping off: identify the pod, namespace, image, node, and pull what the shell did from Falco and the API audit log. Then I think in escalation ladders — did it reach the pod's cloud credentials or node metadata, did it use the service account token against the API server, and is the pod privileged enough to escape to the node. For containment I apply a deny-all NetworkPolicy to isolate the pod and cordon the node, rather than immediately deleting the pod, so I preserve evidence — and I capture process list, connections, and the filesystem first. Then I revoke the pod's identity, and if it may have reached the node or API I treat those as compromised too. Eradicate by redeploying from a clean image and fixing the entry vector.

The key is to capture before the Deployment recreates the pod and erases evidence. First I isolate it with a deny-all ingress/egress NetworkPolicy so it can't do more harm or phone home while I work. Then I snapshot the volatile state — the process tree, network connections, and recently modified files — via kubectl exec, and copy out the relevant filesystem with kubectl cp, recording the exact image ID for comparison against known-good. If I need deeper forensics, I snapshot the underlying node's disk, since the container's writable layer lives there. Only after capture do I delete the pod. Throughout I avoid rebooting or deleting anything that holds evidence, and I note that exec itself is logged and slightly alters the container, so I document what I ran.

That pattern means the token was exfiltrated and is being replayed from outside the cluster. I find which service account and pod it belongs to and enumerate its permissions with kubectl auth can-i --list, because that tells me the blast radius — can it read secrets, create pods, exec. I pull the audit log filtered to that service account to see exactly what it did and from which IPs. For containment I rotate the token by deleting the token secret so the stolen one stops working, set automountServiceAccountToken false going forward, and tighten the SA's RBAC to least privilege. Then I hunt for what it touched and any persistence it created — new bindings, service accounts, or workloads — and rotate any secrets it accessed. Root cause is usually an app compromise that read the mounted token, so I fix that too.

Detection signals include Falco rules for host namespace access or sensitive mounts, an alert on a privileged or hostPath pod, and node-level anomalies like unexpected processes or outbound connections that correlate with a container. Response: once I suspect the node is compromised, I treat the whole node as hostile, not just the pod. I cordon it to stop new scheduling, capture node-level evidence — processes, connections, cron, modified system files, and ideally a disk snapshot — then drain it. Because a container escape means the attacker had root on the node, I assume every pod that ran there and the node's IAM role are compromised, so I rotate the node role and rebuild the node from a clean image rather than trusting it. Recovery is replace, not repair.

The clearest is creation or modification of RBAC that grants broad power — a new ClusterRoleBinding to cluster-admin, or RoleBindings handing a service account elevated verbs. Also: use of the impersonate verb, create on pods with privileged or hostPath specs, create on pods/exec into sensitive pods, and bulk get on secrets. I'd look at the verb, the resource, the user or service account, and the source IP together — for instance a service account that normally only reads its own namespace suddenly creating cluster-wide bindings from an external IP. The audit log's RequestResponse level on RBAC and secrets is what makes these visible, which is why configuring the audit policy to capture them in advance matters.

I separate network containment from destruction. I can contain fast without losing evidence by applying a deny-all NetworkPolicy to isolate the pod and cordoning the node — that stops C2, exfiltration, and lateral movement immediately while the pod stays alive for capture. What destroys evidence is deleting the pod, which triggers recreation, so I do that only after snapshotting processes, connections, filesystem, and ideally the node disk. The exception is active harm — if it's exfiltrating data or spreading right now, containment wins and I accept evidence loss. And I scope before I broadly remediate, because revoking one thing prematurely can tip off the attacker before I've mapped the blast radius. So the rule is isolate-then-capture-then-eradicate, with active-harm as the override.