Approaching Open-Ended Security System Design in Technical Interviews

#Start here: the basics in plain words

An open-ended design question is one with no answer key. "Design a platform that runs untrusted code." "Secure our build cluster." "We want engineers to SSH into production; make it safe." The interviewer has a system in mind, usually one they operate, and wants to watch you turn a vague sentence into a defensible architecture in under an hour, out loud, while they poke at it.

You are not being graded on the final diagram. You are being graded on the process: whether you ask before you build, whether you can draw a trust boundary and say what crosses it, whether you rank threats instead of listing them, whether you name the cost of each control, and whether you can be told "that won't work" without falling apart.

Memory hook

scope, sketch, boundaries, threats, controls, costs, summary. Seven words, roughly eight minutes each. If you can say where you are in that list at any moment, you are in control of the interview.

#What is this?

A method for the hour: eight phases with timings, the questions to ask in each, what to say out loud, and how to recover when the interviewer challenges you. It is deliberately mechanical. Under pressure, mechanical is what survives.

#Why it matters

Candidates with strong knowledge fail these interviews in predictable ways: they start listing controls before they know the system, they draw every box but no boundaries, they treat the first pushback as a verdict, or they run out of time with no summary. Interviewers at companies that run untrusted code at scale (CI platforms, code-execution sandboxes, agent runtimes) use exactly this format because the job is exactly this: ambiguous requirements, hostile input, real hardware, no answer key.

#The framework

#Phase 1: Clarify the prompt and define scope

What is this?The first five minutes decide whether you design the right system. The prompt is deliberately vague; the interviewer has answers to questions you have not asked yet.

How it worksRestate the prompt in one sentence that names the untrusted input and the valuable thing it is near. Then ask only questions whose answers would change your design. Three or four is right; eight is stalling.

The question bank, grouped by what each answer changes:

In practiceWhat you would say, for the prompt "secure our multi-tenant build cluster that runs untrusted code":

"Let me restate: we run other people's code, including strangers' pull requests, on shared hardware that also
has routes to things we care about. I want to bound what a job can reach, keep and impersonate.
Before I draw anything:
  1. Are the workloads from GitHub Actions and Buildkite, or our own API too?           ← token model
  2. Do tenants share hosts concurrently, or can I dedicate?                              ← sandbox tier, side channels
  3. Do jobs need to reach anything private: our network, customer VPCs, cloud APIs?      ← egress and identity
  4. What is the hardware: Linux racks, cloud, Apple Silicon for macOS builds?             ← virtualisation choices
I'll assume answers where you don't have them and say so."

🎯

On the job. This is the same conversation you have at the start of a real design review. The engineer who asks "who can submit a job?" before "which sandbox?" is the one whose design survives contact with the product.

#Phase 2: Assumptions and constraints

What is this?Whatever the interviewer did not answer, you decide, out loud, so your design can be judged against your assumptions rather than against theirs.

How it worksState the baseline in one breath: the hardware, the scale, the trust model, the budget for latency. Write it in the corner of the board; you will point at it later when you defend a trade-off.

In practice

Assumptions (say them, write them):
  hardware   bare-metal Linux racks (AMD EPYC x86-64, AmpereOne arm64), Apple Silicon hosts for macOS via Apple's vz
             framework (two guests per host max), burst capacity in AWS and GCP
  scale      ~1M jobs/day, mean 6 min, p95 30 min → ~4k concurrent, plan 12k burst
  trust      every job is hostile; a PR from a fork is a stranger's code with root inside its sandbox
  must reach public registries, GitHub, optionally a customer-declared allowlist; never our internals or cloud metadata
  budget     sandbox start under 1 s on Linux; macOS pooled; 5–10 % CPU overhead acceptable for isolation
  failure    security controls fail closed; a down egress proxy means jobs wait, not jobs bypass

If the interviewer corrects an assumption, thank them and change it; that is the system working.

#Phase 3: Baseline design

What is this?The boring version of the system, drawn before any hardening, so there is something to put boundaries on. Candidates who skip this end up hardening boxes that do not exist yet.

How it worksSeparate the control plane (what you trust: API, scheduler, provisioner, token broker, audit sink) from the data plane (what runs untrusted code: hosts, sandboxes, their network). Then narrate one job's life from event to destruction. Name the real components: a Kubernetes cluster or a host agent on bare metal, Firecracker or gVisor or a vz guest per job, BuildKit for image builds, a Go proxy for egress, Buildkite or GitHub Actions as the job source.

         control plane (trusted)                       data plane (assume compromised)
   webhook → scheduler → provisioner → token broker    host agent → sandbox per job → egress proxy → internet
                                   └→ audit/provenance sink  ◄──────── logs, write-only from jobs

Say the lifecycle in eight steps: event, schedule, mint identity, provision, run, egress, collect, destroy. Each step is a place a boundary will go in the next phase.

In practiceThe sentence that shows you know what you drew:

"Everything left of this line never executes customer code. Everything right of it I assume is compromised at
any given moment, so every arrow from right to left is a boundary I'll come back to.   ← the interviewer now knows your model

#Phase 4: Trust boundaries and adversarial threat modelling

What is this?The phase that separates security design from system design. You draw the lines where trust changes and walk attacks across each one, producing a ranked table rather than a brainstorm.

How it works

In practiceThe table you build on the board, for the build cluster:

 #  attacker → target             path                                   blast radius        rank
 1  job → host kernel             syscall bug, runc/BuildKit escape      every tenant on host  1
 2  job → control plane           network route + auth bug               whole platform        1
 3  job → other tenant            shared cache, shared segment, SMT      one tenant            2
 4  job → cloud account           IMDS from a cloud-burst sandbox        cloud account         2
 5  job → customer cloud          over-broad OIDC trust policy           one customer          3
 6  PR job → release build        poisoned cache restored by trusted job one tenant's releases 2
 7  human → node                  break-glass shell lands on the host    every tenant on host  2
 8  platform → customer           we alter or misattribute artifacts     reputation, all       3

The ranking is the point. Two boundaries get the deep dives; the rest get a sentence each.

📰

Real incident — Capital One (2019). The pieces you need to know. A web application firewall ran on a cloud virtual machine. Cloud VMs can ask a local metadata service, at a fixed link-local address, for temporary credentials belonging to the VM's assigned role. Server-side request forgery (SSRF) is when an attacker makes a server fetch a URL of the attacker's choosing. What went wrong. A misconfiguration let an outsider make the firewall host fetch the metadata address. The response was the role's credentials; the role had permission to list and read storage buckets. Roughly a hundred million customer records were copied out. Why it's bad. Nothing was "hacked" in the exploit sense. A request crossed one trust boundary (internet to firewall) and was relayed across a second (host to metadata service) that nobody had drawn, and the credential on the far side was far broader than the host needed. The lesson for this interview. When you draw "sandbox to network", ask what else is on that network. The metadata service is the classic undrawn boundary; a job in cloud burst capacity that can reach it has the instance's identity. Draw it, block it at the datapath, give the instance no useful role, and require the metadata service's hop-limited mode so the answer cannot be forwarded.

🎯

On the job. The threat table is the artefact that outlives the meeting. In a real review it becomes the list of controls with owners; in the interview it is the thing you point at when the interviewer asks "why that first?" and you answer "because row one is every tenant on the host".

#Phase 5: Technical controls and defence in depth

What is this?Now, and only now, the controls. Ranked by the table, with two or three explored deeply and the rest named so the interviewer knows you know they exist.

How it worksFor each control say three things: what it bounds, what layer it lives at, and what it costs. Depth is a second sentence about mechanism; a third sentence about how it fails.

In practiceGoing deep on one, the way the interviewer wants to hear it:

"Row one first: job to host kernel. The control is a VM per job. On the EPYC and AmpereOne hosts that's a
Firecracker microVM booted from a read-only golden image with a per-job scratch disk and a per-job tap device.
Mechanism: the job's syscalls hit the guest kernel; reaching the host means a second bug in the VMM or KVM,
and Firecracker's jailer runs the VMM as an unprivileged user in a chroot with seccomp, so even that is bounded.
Failure: if someone ships a privileged pod or a hostPath for convenience, the wall is gone, so admission policy
on the cluster rejects those outright; and if the VMM has a bug we treat the host as compromised, drain it,
and re-image, which is why hosts are immutable in the first place.
Cost: a few percent of CPU, more for small-file I/O; I'd offer bigger VMs rather than weaker isolation."

That is four sentences, each doing a job: what, how, how it fails, what it costs.

#Phase 6: Trade-offs, failure modes and blast radius

What is this?The phase that proves you have operated something. Every control you named has a cost and a way to break; say both before you are asked.

How it worksWalk your controls and for each ask "what happens when this is down, and what happens when this is wrong?"

Then the two sentences every interviewer waits for: "This fails closed, and here is what that costs." And: "The blast radius of the worst realistic failure is one tenant's jobs on one host for the minutes until drain, not the fleet."

🎯

On the job. The real trade-off meetings are about hit rates and startup latency, not kernels. A design whose only argument is "it's more secure" loses to the team that ships faster. Bring the number: "a VM per job costs us 6 % CPU and 300 ms start; a shared kernel costs us every tenant on the host when runc has its next CVE."

#Phase 7: Narrative control and thinking out loud

What is this?The meta-skill: making your process visible so the interviewer can follow it, steer it and credit it.

How it works

In practicePhrases that do the work:

"Let me restate where we are: scope and assumptions done, baseline drawn, I've ranked eight threats and I'm
about to go deep on the top two. Shout if you'd rather I take a different one."
"I'm going to make a call here and move on: VM per job. I'll justify it in trade-offs."
"Parking that; it's a good question and it changes the macOS story. Back to egress."

#Phase 8: Recovering when stuck or challenged

What is this?The interviewer will push: "that won't work", "we tried that", "what if the proxy is compromised?". This is not a verdict. It is the part of the interview where they find out whether you can debug your own design.

How it worksThe move is always the same: find the assumption the challenge breaks, say it, and redesign from there.

In practiceA real exchange, and the recovery:

Interviewer: "Your egress proxy does hostname allowlists. A build runs `npm install`, and a dependency's
              postinstall script talks to a CDN that also hosts the attacker's bucket. Now what?"

You:  "Right: the allowlist is by host name, and a shared CDN host name can front both. So host-name allowlists
       are not a complete control for CDN-hosted destinations; I had assumed registries were distinct hosts.
       Two moves. First, for the registries we care about we mirror them internally, so the allowlist points
       at our mirror and the CDN isn't needed at build time. Second, where the CDN is unavoidable I treat
       it as a logged, rate-limited path and rely on the next layer: the credentials in that job are minutes
       long and scoped to that job, so what leaks is worth little. I'll add 'shared CDN hostnames' as a row
       in the threat table with 'mirror plus short-lived credentials' as the control."    ← assumption named, control adjusted, table updated

🎯

On the job. This exact loop is a production incident review: a control was assumed to cover a case, it did not, you name the gap, you add a layer. Candidates who can do it calmly in the interview are the ones who do it calmly at three in the morning.

#The hour, compressed: a worked example

The prompt: "Secure a heterogeneous, multi-tenant build cluster running untrusted arbitrary code." The full design is on the multi-tenant CI runner platform page; this is the shape of the hour.

In practiceThe summary you end on:

"Three invariants. A job can reach only the proxy and its allowlist. A job keeps nothing: VM, disk, tap,
tokens and cache credentials are per-job and destroyed together, verified by the host agent. A job can
pretend to be nothing but itself: single-use registration, OIDC pinned to immutable ids, provenance signed
by the controller. Build order: isolation and immutable hosts, then egress, then the token broker, then
caches, then break-glass, then provenance. If you only let me ship two, it's the first two."

#What interviewers actually score

Not a published rubric, but what the feedback forms tend to ask:

🎯

On the job. The same eight items are a good self-review for a design document. If a doc has no explicit boundaries, no ranked threats and no costs, it is a list of tools, and it will be read as one.