Mock Interview: Threat-Model a Privileged Node Agent
DifficultyMedium–Hard | Format: 60-minute open problem with one engineer, scored on how you think rather than on what you remember
Problem Statement
"We run customers' CI jobs on bare-metal machines. Each machine is a Kubernetes node. On every node we run our own agent as a DaemonSet with
hostPID: trueandhostNetwork: true. It collects metrics, ships logs and lets support engineers debug stuck jobs. Threat-model it and tell me how you'd harden it."
This page is a worked hour: what to say, when, and why. The same rhythm works for any open problem: a cache shared between customers, an egress firewall, SSH into a running job.
ImportantThe interviewer is not waiting for a list of controls. They want to see you scope, draw the boundaries, reason about who attacks whom, rank what matters, and change your mind when they give you new facts. Say your reasoning out loud the whole time.
What is this?
A node agent is a program that runs once on every machine in a cluster, usually as a Kubernetes DaemonSet: a workload type that puts exactly one copy on each node. Agents like this collect metrics, ship logs, enforce network rules or run security sensors. To do that they often ask for host-level access:
hostPID: truethe pod shares the machine's process list. Inside it, ps aux shows every process on the node, including other customers' jobs.
hostNetwork: truethe pod uses the machine's own network stack. It sees every interface, listens on the node's IP, and reaches anything the node can reach.
Why it matters
The agent sits on every machine next to untrusted code. If it's compromised, one bug becomes access to the whole fleet. And because it reads things that customers control (process names, log lines, files inside their containers), the customer can attack it, not only the other way round. Interviewers like this problem because there's no single right answer: it's all trade-offs between what the agent needs and what it exposes.
How the hour goes
-
0–5 min · Restate and clarifybrowserdon't solve yet
- Repeat the problem in one sentence
- Ask 3–4 questions that change the design
-
5–12 min · Assumptions and a sketchdnsdraw the node
- Say your assumptions out loud
- Draw kernel, kubelet, runtime, customer jobs, agent
-
12–25 min · Threat modelattackboth directions
- Agent compromised → what the attacker gets
- Customer attacks the agent → confused deputy
-
25–40 min · Mitigations, rankeddefendbiggest risk first
- Remove the need before restricting it
-
40–52 min · Go deep where they pushserverdetection, cost, edge cases
-
52–60 min · Summary and your questionsrender
0–5 min: restate and clarify
Restate it in one sentence, then ask the questions whose answers change the design. Don't ask questions for the sake of asking.
In practiceWhat you'd actually say:
"So: a privileged agent on every node, next to untrusted customer jobs. I want to know what
it can reach and who can reach it. Four quick questions before I start:
1. Do customer jobs run as containers directly on the node, or inside VMs on the node? ← changes everything
2. What does the agent actually need hostPID and hostNetwork for?
3. Does it run as root, and with which Linux capabilities?
4. Who can talk to it: is there an API, and do support engineers reach it from outside?"Memory hook"Where does the customer's code run?" Always ask it first. If jobs run in VMs,
hostPIDonly shows the VM processes (QEMU or Firecracker), not what runs inside them. If jobs run as plain containers,hostPIDshows the customer's processes directly.
5–12 min: assumptions and the sketch
If the interviewer says "assume containers, the agent runs as root, it has a small HTTP API", say your assumptions out loud and draw:
┌──────────────────────────── bare-metal node ─────────────────────────────┐ │ Linux kernel (one kernel shared by everything below) │ │ │ │ kubelet ──── containerd │ │ │ │ │ ┌───────────┼──────────────┬───────────────────────────┐ │ │ [job A] [job B] [node agent] hostPID + hostNetwork, root │ │ customer 1 customer 2 │ reads /proc, cgroups, container logs │ │ │ HTTP API on :9100 ← support engineers │ │ └── ServiceAccount token (Kubernetes API) │ │ Node network: management network, cloud metadata, BMC (out-of-band) │ └───────────────────────────────────────────────────────────────────────────┘ Trust boundaries: customer ↔ customer, customer ↔ agent, agent ↔ node, node ↔ control plane
TipNaming the trust boundaries out loud is what tells the interviewer you think in systems. Point at each line on the drawing as you say it.
12–25 min: threat model in both directions
Direction 1: the agent is compromisedThrough a bug in it, a malicious image update or a stolen support credential. What does the attacker get?
hostPIDsees every process. As root it can read /proc/<pid>/environ (environment variables, often secrets) of processes it may ptrace-read. The kernel allows that when the user IDs match and the caller's capabilities are a superset of the target's, or when the caller has CAP_SYS_PTRACE. Other containers running as root with the default capability set pass that check. So the agent can read other customers' secrets and their filesystems through /proc/<pid>/root/. With CAP_KILL (in the default set) it can also kill their processes.
hostPID + CAP_SYS_PTRACE or privilegednsenter -t 1 -a gives a shell on the host itself. That's a full node compromise.
hostNetworkreaches everything the node reaches: the management network, cloud metadata or the BMC (the out-of-band server controller), node-local services such as the kubelet. Most network plugins don't apply NetworkPolicy to host-network pods. With CAP_NET_RAW, which many runtimes grant by default, it can capture traffic on every interface, including other pods' virtual interfaces.
its RBAC permissions, from every node. If it can read Secrets or create pods cluster-wide, one node becomes the whole cluster.
so one bad image push is a fleet-wide compromise.
In practiceWhat hostPID looks like from inside the agent pod:
$ ps -eo pid,user,comm | head
PID USER COMMAND
1 root systemd ← the host's init: you are in the host PID namespace
812 root kubelet
4411 1001 cargo ← customer A's build
5120 root node ← customer B's build, running as root
$ cat /proc/5120/environ | tr '\0' '\n' | grep -i token
NPM_TOKEN=npm_9f2c... ← another customer's secretDirection 2: a customer attacks the agent. This is the part most candidates miss. The agent is privileged, and it reads things the customer controls:
parsing bugs in the agent.
injection into the log pipeline or into a support engineer's terminal (escape sequences).
if the agent follows a path like /proc/<pid>/root/var/log/app.log, the customer can replace it with a symlink to /etc/shadow on the host. The privileged agent reads the host file for them. That's a confused deputy: the customer borrows the agent's privilege.
if the agent's HTTP API listens on the node IP, every job on the node can call it.
📰Real incident — Kubernetes subPath symlink escape (CVE-2021-25741). A user could create a symlink inside their volume so that the kubelet, running with full privilege, mounted a host path into their container. It's the same pattern: privileged code following a path the user controls. The lesson: privileged components must open user-controlled paths without following symlinks (
openat2withRESOLVE_NO_SYMLINKS/RESOLVE_BENEATH), or not open them at all.
📰Real incident — Azurescape (2021). Researchers broke out of a container in Azure Container Instances through an old runc bug (CVE-2019-5736). They found the node's privileged components reachable from there and took over other customers' containers in the shared cluster. It's the fleet-wide picture: one node escape plus over-trusted node components equals crossing between tenants.
25–40 min: mitigations, biggest risk first
Say why each one ranks where it does. Start by removing the need for privilege, then restrict what's left.
Then hostPID on the node only sees hypervisor processes, and the shared kernel stops being the boundary. This is the biggest single change; say so and say what it costs (boot time, memory overhead).
A tiny privileged part does only the few things that need the host (read cgroup stats, attach an eBPF probe). The large unprivileged part does parsing, shipping and the API, and talks to the small part over a Unix socket. Bugs in parsing no longer run as root.
Drop all capabilities and add back only the named ones; there should be no SYS_PTRACE and no privileged. Use a read-only root filesystem, the runtime's default seccomp profile and no NET_RAW unless it captures packets.
Open files with openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS) or read through the container runtime's API instead of /proc/<pid>/root.
The agent API listens on a Unix socket or localhost and requires authentication. Block the management network, metadata and BMC from the pod with host firewall rules (eBPF or nftables), not NetworkPolicy, which doesn't apply here.
Give it read-only access to what one node needs, use short-lived tokens bound to the pod, and never let it read Secrets cluster-wide.
Use Pod Security Admission: restricted for customer namespaces, privileged only for the system namespace. Add an admission rule that only this signed image and this ServiceAccount may set hostPID.
Alert when the agent does what it shouldn't: a shell spawned from it, reads of /proc/*/environ, or connections to the management network.
In practiceThe admission rule from point 7, as a ValidatingAdmissionPolicy (built into Kubernetes, rules written in CEL):
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: host-namespaces-only-for-node-agent }
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- { apiGroups: [""], apiVersions: ["v1"], operations: ["CREATE", "UPDATE"], resources: ["pods"] }
validations:
- expression: >-
!(has(object.spec.hostPID) && object.spec.hostPID ||
has(object.spec.hostNetwork) && object.spec.hostNetwork) ||
(object.metadata.namespace == "node-system" &&
object.spec.serviceAccountName == "node-agent") # ← the only exception
message: "hostPID/hostNetwork are reserved for the node agent"🎯On the job. In a review of a DaemonSet manifest, these are the four lines that matter:
hostPID,hostNetwork,privileged/capabilities.add, andhostPathvolumes. For each, ask "what for?" and "what breaks if I remove it?" Often the answer is "nothing; it was copied from an example".
40–52 min: where they will push
| They ask | Good direction |
|---|---|
| "We really need hostPID for per-process metrics." | Read cgroup v2 files (/sys/fs/cgroup/.../cpu.stat) or use eBPF from the small privileged part. You get per-container numbers without seeing every process. |
| "How would you know if the agent was abused?" | Behaviour-based rules on the node: process tree of the agent, file reads under /proc/*/environ, unexpected outbound connections. Send the alerts off the node so a compromised node can't delete them. |
| "What does all this cost?" | VMs: boot time and memory. Split agent: more code to maintain. Admission policy: almost free. Say the order you'd ship them in. |
| "Support engineers need a shell on stuck jobs." | Give them a shell into that one job's sandbox, not the node. Use short-lived credentials tied to a person, record the session, and close it when the job ends. |
| "A customer runs as root inside their container. Does that change anything?" | Yes. Without user namespaces, root in the container is UID 0 on the host. That makes the ptrace check and file permissions much weaker, so isolation has to come from VMs or user namespaces. |
When you don't knowSay so, then reason: "I'm not sure whether your network plugin applies policy to host-network pods. I'd check, but I'll assume it doesn't and enforce on the host instead." That's a strong answer; guessing confidently isn't.
52–60 min: summary and questions
Close in three sentences: the biggest risk, the first two things you'd do and what you'd measure. Then ask questions that show you were thinking about their system, e.g. "Do your customer jobs share a kernel with anything you run?" or "What does your agent need that a VM boundary would take away?"
Security angle
while customer jobs run as containers. Every host-level option weakens it.
Parsing, path-following and log rendering all run with the agent's privilege.
Image signing, a staged rollout and an admission allowlist limit how far a bad version can spread.
Enforce on the host or don't grant it.
Interview Q&A
hostPID: true let a pod do?The pod joins the host's process ID namespace, so it sees every process on the node: the host's own services and every other pod's processes. Seeing them is only the start. As the same user and with at least the target's capabilities, it can read their /proc/<pid>/environ, which often holds secrets, and reach their files through /proc/<pid>/root. With the default CAP_KILL it can signal them. If it also has CAP_SYS_PTRACE or runs privileged, nsenter -t 1 gives a shell on the host. So on a node running other customers' containers, hostPID plus root is close to reading every tenant's secrets.
hostNetwork pod, and what do you do instead?NetworkPolicy is enforced on a pod's own network interface. A host-network pod has no interface of its own: it uses the node's, so most network plugins can't tell its traffic from the node's and don't apply pod policy. It can reach whatever the node reaches: the management network, metadata, BMCs, kubelet ports. I'd enforce on the host instead (nftables or eBPF rules attached to that pod's cgroup), and keep anything sensitive off the networks the node can route to.
It's when an unprivileged user gets a privileged program to act for them. A node agent running as root that reads a customer's log file through /proc/<pid>/root/... can be tricked by a symlink into reading a host file such as /etc/shadow. The kubelet bug CVE-2021-25741 was the same pattern with volume subPaths. Prevention: don't open user-controlled paths from privileged code. If you must, use openat2 with RESOLVE_BENEATH and RESOLVE_NO_SYMLINKS, or ask the container runtime for the data, and do parsing in an unprivileged process.
Split it. A small privileged component does only what needs the host: reading cgroup stats, attaching eBPF probes, maybe a few syscalls. It exposes a narrow API over a Unix socket. Everything else, parsing, batching, shipping and the support API, runs unprivileged in its own container with a read-only filesystem and no capabilities. Most bugs live in the big part, and now they don't run as root. It's the same idea as privilege separation in OpenSSH.
hostPID on these nodes?Two layers. Pod Security Admission: label customer namespaces restricted, which forbids host namespaces, privileged mode and host paths. On top, a ValidatingAdmissionPolicy (or Kyverno/Gatekeeper) that allows hostPID and hostNetwork only for one namespace, one ServiceAccount and one signed image. I'd also alert on any pod that sets them, because admission rules can be changed by someone with cluster-admin.
Run each job in a lightweight VM (Firecracker, Cloud Hypervisor or Kata) so the boundary between customers is a hypervisor, not a shared kernel. Then a host-level agent sees hypervisor processes, not customer processes, and a kernel bug inside a job stays inside its VM. The cost is boot time (usually well under a second for microVMs), memory per VM, and more complex networking and storage. For CI, where each job is short and untrusted by definition, that cost is usually worth it. I'd measure job start-up time before and after.
Know its normal behaviour and alert on anything else: child processes (a shell from the agent is never normal), reads of /proc/*/environ, writes outside its own directories, and connections other than to the metrics and log endpoints. Collect that with an eBPF sensor and send it off the node immediately, so an attacker with root on the node can't delete the evidence. Compare the running image digest against the signed one at deploy time and periodically.
Give them access to the job, not the machine: a shell inside that job's sandbox through a broker. They authenticate as themselves with SSO, the broker issues a short-lived certificate valid only for that job, and the session is recorded and linked to a ticket. Access ends when the job ends. The node agent never needs a remote shell API, which removes the biggest single risk from it.