System Design: Hardening and Architecting a Secure Multi-Tenant CI Runner Platform

#Problem Statement

"Design a CI platform that runs customers' build jobs. Customers push code, including pull requests from strangers, and we run it. Jobs need root, Docker builds and the internet. We own bare-metal Linux racks (AMD EPYC and AmpereOne) and Apple Silicon machines for macOS builds, plus some capacity in AWS and GCP. Make it safe for us and for the other customers on the same hardware."

This page is a worked hour: what to say, in what order, and why. The isolation internals (how gVisor and microVMs differ, how escapes happen) live on the sandbox page; this page is about the system: control plane, token lifecycle, caches, egress, break-glass, provenance, and how the parts fail.

Important

The interviewer wants to see you treat the platform as what it is: remote code execution as a service, sold to strangers, next to other customers' secrets, with a network route to your internals. Every design decision should trace back to one of three questions: what can a job reach, what can a job keep, and what can a job pretend to be.

#Start here: the basics in plain words

Memory hook

reach, keep, pretendEvery control on this page limits one of them. Sandboxing and egress limit what a job can reach. Ephemeral VMs and partitioned caches limit what a job can keep. Single-use tokens and OIDC claims limit what a job can pretend to be.

#What is this?

A multi-tenant CI runner platform is a fleet of machines that accept jobs from many customers' pipelines, execute them in isolation, give them controlled access to the network and to credentials, collect the outputs, and leave nothing behind. The hard part is that the workload is maximally privileged (root, nested containers, arbitrary network) and maximally untrusted (anyone can open a pull request), and the business wants it to be fast and cheap.

#Why it matters

Interviewers use this problem because it touches every layer: hypervisor and kernel isolation, container internals, identity federation, network design, storage integrity, observability and incident response, with real trade-offs between security and the developer experience that pays for it. It is also the shape of several real companies, which is why the stack hints (Buildkite and GitHub Actions, gVisor and BuildKit, Apple Silicon with vz, EPYC and AmpereOne racks, Go services, OIDC) keep appearing.

#How the hour goes

#0–5 min: Restate and clarify

Restate in one sentence, then ask questions that change the design. Not "what language is the control plane in"; that changes nothing.

In practiceWhat you would actually say:

"So: we execute untrusted code from many customers on shared hardware, and the code expects root,
Docker and internet. I want to bound what a job can reach, keep and pretend to be. Four questions:

1. Do jobs come from GitHub Actions and Buildkite, or also our own API?   ← decides the token model
2. Is the hardware shared between tenants at the same time, or can I dedicate hosts?   ← decides sandbox tier
3. What must jobs reach: public registries only, or customer VPCs and private registries too?   ← decides egress
4. What are the macOS constraints: Apple's two-VMs-per-host rule, and do customers need their own images?"

Assume answers: both CI services plus a thin API of your own; hardware is shared, dedicated hosts are a premium option; public registries plus an optional customer-provided allowlist; macOS is for mobile builds with your images.

#5–12 min: Requirements, assumptions and numbers

FunctionalRun a job to completion with logs streamed back; support Linux x86-64, Linux arm64 and macOS arm64; docker build inside jobs; per-job secrets from the CI service; caching to make repeat builds fast; a way for a developer to debug a stuck job.

Security requirements, stated as invariants you will defend later:

• A job cannot read or write anything of another tenant's: files, caches, network traffic, credentials.
• A job cannot persist anything on the host across jobs.
• A job cannot reach the control plane, host management interfaces, cloud metadata services or RFC 1918 space except through explicit, logged allowlists.
• A credential issued to a job is useless after the job ends and useless from anywhere else.
• Everything the platform does to a job is attributable and logged outside the job's reach.

Numbers, so the design has a scale. Say them as assumptions:

jobs/day            ~1,000,000            (assumption; say it out loud)
mean duration       ~6 min, p95 30 min
concurrency         1e6 × 6 min / 1440 min ≈ 4,200 jobs running at once; plan for 3× burst ≈ 12,000
Linux host          EPYC 128 cores / AmpereOne 192 cores → ~30–60 sandboxes per host at 2–4 vCPU each
macOS host          2 macOS guests per machine (Apple's licence and the framework cap) → macOS is capacity-bound, not CPU-bound
cold sandbox start  Firecracker microVM ~150 ms to 1 s; gVisor pod ~0.5–1 s; macOS vz guest from a template 20–60 s

The last line shapes the macOS design: you pre-boot and pool macOS guests, which means the "fresh VM per job" invariant needs a reset story, not just a boot story.

#12–22 min: Baseline architecture

Draw it before hardening it. Control plane on the left, data plane on the right, and arrows that only go one way where you can manage it.

                        CONTROL PLANE (trusted; never runs customer code)
 ┌──────────────────────────────────────────────────────────────────────────────────────────┐
 │  Webhook/API ──► Scheduler ──► Provisioner ──► Token broker ──► Attestation / audit sink  │
 │  (GitHub app,     (queue per      (boots a         (mints single-use    (signs provenance, │
 │   Buildkite        tenant, host    sandbox on a     runner regs, job     stores logs where │
 │   cluster API,     picking,        chosen host,     OIDC, cache creds)   jobs can't reach) │
 │   own API)         quotas)         injects config)                                         │
 └───────────┬──────────────────────────┬─────────────────────────────────┬──────────────────┘
             │ gRPC, mTLS, SPIFFE ids   │ host agent API (one-way: control plane → host)       │
             ▼                          ▼                                 ▼
 ═══════════ DATA PLANE (assume compromised) ══════════════════════════════════════════════════
  Linux rack (EPYC / AmpereOne)                 Apple Silicon rack                 Cloud burst (AWS/GCP)
  ┌────────────────────────────────┐          ┌────────────────────────┐        ┌─────────────────────┐
  │ host OS (minimal, immutable)   │          │ macOS host             │        │ bare-metal or nitro │
  │  host agent (Go)               │          │  host agent (Go)       │        │  instances only;    │
  │  ┌────────┐ ┌────────┐         │          │  ┌─────────┐ ┌───────┐ │        │  IMDS firewalled    │
  │  │ microVM│ │ microVM│  …×40   │          │  │ vz guest│ │vz gst │ │        │  from guests        │
  │  │ runner │ │ runner │         │          │  │ runner  │ │runner │ │        └─────────────────────┘
  │  │ job    │ │ job    │         │          │  │ job     │ │job    │ │
  │  └───┬────┘ └───┬────┘         │          │  └────┬────┘ └───┬───┘ │
  │      └─────┬────┘ tap/bridge   │          │       └────┬─────┘     │
  └────────────┼───────────────────┘          └────────────┼───────────┘
               ▼                                           ▼
        ┌──────────────── egress tier ────────────────────────────┐
        │ per-job policy proxy (Go)  → allowlisted hosts only      │
        │ dedicated egress IP pools per tenant (for their allowlists)│
        │ default-deny to RFC1918, 169.254.0.0/16, control plane    │
        └────────────────────────┬─────────────────────────────────┘
                                 ▼
                     internet: registries, GitHub, customer endpoints
   
  Side services (reached only via the proxy or a per-job credential):
   cache store (per-tenant buckets)   artifact store   package mirrors   log/attestation sink (write-only from jobs)

The job lifecycle, before hardening

#22–32 min: Threat model

Draw the boundaries on the diagram, then walk the attacks across each one. Say the attacker and the target before the mechanism.

🎯

On the job. T4 is the one candidates forget and attackers love. If a sandbox can open a TCP connection to the scheduler's gRPC port, every other control is one authentication bug away from irrelevant. Make the host agent call out to the control plane; never let the control plane be reachable from the job network.

📰

Real incident — Codecov Bash Uploader (2021). The pieces you need to know. Many CI jobs ended with curl -s https://codecov.io/bash | bash to upload coverage. The job's environment variables, including deploy credentials, were in that process's reach. Codecov built its Docker images in a way that left a cloud storage credential inside an image layer. What went wrong. Someone extracted that credential, used it to edit the hosted script, and added one line that posted the output of env and the git remote URL to an outside server. The change sat there from late January to April 2021, until a customer compared the script's checksum with the published one and saw a mismatch. Why it's bad. Every pipeline that ran the script handed its secrets to the attacker, with nothing on the pipeline side changing. The exfiltration was an ordinary HTTPS POST from a job that was allowed to talk to the internet. The lesson. For this design: a per-job egress allowlist would have turned the POST into a logged deny, since the attacker's host was not a registry or a Codecov domain; short-lived OIDC credentials would have made the stolen env worthless an hour later; and the platform should make checksum-verified fetches easy so customers stop piping the internet into bash.

#32–48 min: Deep dives, ranked by blast radius

Pick the order by what a compromise costs. Kernel escape first (T1, T2), then identity (T4, T6, T8), then egress (T5), then caches (T7), then the human paths (break-glass) and the proof (provenance).

#1. Isolation tiers per hardware class

Decision tableSay the row, then the reason.

Why not plain containers: a container shares the host kernel, so a job's syscalls hit the same code that protects every other tenant. Why not gVisor alone: it narrows the host kernel surface dramatically but is still one Go process on the host; an escape from the Sentry is an escape to the host. Why gVisor inside the VM for Docker builds: BuildKit's own history of escapes (the January 2024 set, CVE-2024-23651 and CVE-2024-23653, and runc's CVE-2024-21626) means the build step is the most likely thing to break out of its container; giving it a second wall inside a sacrificial VM makes that a nuisance rather than an incident. The sandbox page covers the mechanics; here you say the decision and the reason.

In practiceWhat the host agent enforces at VM creation, abridged from a Firecracker config the controller generates:

{
  "boot-source": { "kernel_image_path": "/images/vmlinux-6.12-hardened", "boot_args": "console=ttyS0 reboot=k panic=1 pci=off init=/sbin/runner-init" },
  "drives": [
    { "drive_id": "rootfs", "path_on_host": "/images/runner-linux-2026.10.3.ext4", "is_root_device": true, "is_read_only": true },   ← golden image, read-only
    { "drive_id": "scratch", "path_on_host": "/var/lib/jobs/j-8f3a/scratch.ext4", "is_root_device": false, "is_read_only": false }  ← per-job, discarded after
  ],
  "network-interfaces": [ { "iface_id": "eth0", "host_dev_name": "tap-j8f3a", "guest_mac": "02:00:00:8f:3a:01" } ],   ← one tap per job; host routes it only to the proxy
  "machine-config": { "vcpu_count": 4, "mem_size_mib": 8192, "smt": false },                                           ← no sibling-thread sharing with another tenant
  "mmds-config": { "version": "V2", "network_interfaces": ["eth0"] }                                                   ← job config via MMDS, not a cloud IMDS
}

#2. Image builds without a privileged daemon

What is this?Customers run docker build. The naive approach is Docker-in-Docker: a privileged container running dockerd. Privileged means all capabilities, no seccomp, device access: a container escape is a configuration, not an exploit.

How it works insteadRootless BuildKit: buildkitd runs as an unprivileged user inside a user namespace (the kernel feature that maps "root" inside to an ordinary UID outside), using rootlesskit for its own mount and network namespaces. Builds are launched daemonless with buildctl talking over a Unix socket that only this job can see. Inside a microVM this is belt and braces: even a full rootless-BuildKit escape lands you as a non-root user in a guest that is destroyed in minutes.

# inside the job VM: what "docker build" is actually running
$ ps -o user,pid,cmd -u builder | head
USER       PID CMD
builder   1193 rootlesskit --net=slirp4netns --disable-host-loopback buildkitd --oci-worker-no-process-sandbox   ← not root; no host loopback, so no reaching host services
builder   1201 buildkitd ...
$ cat /proc/1201/uid_map
         0       1000      65536                                        ← "root" in the build namespace is UID 1000 outside
$ buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. \
    --opt network=none \                                                  ← RUN steps get no network; dependencies come from the mirror in a fetch stage
    --export-cache type=s3,bucket=cache-tenant-acme,prefix=repo-42/trusted/ \   ← tenant- and trust-scoped cache
    --output type=image,name=registry.internal/acme/app:sha-e1b2,push=true

Cache storage for BuildKit layers is per tenant and keyed by the trust class of the build (more below); a tenant's rootless BuildKit never mounts another tenant's cache directory because the directory does not exist inside its VM.

#3. Runner registration tokens: the complete lifecycle

What is this?A runner has to prove to GitHub or Buildkite that it is allowed to take jobs. The long-lived way is a registration token or agent token in a config file, which, on a multi-tenant platform, means every job can read a credential that registers new runners.

How it worksBoth services support an ephemeral model; the control plane owns the credential, the job never sees it.

Zero cross-job state remanence is a property you verify, not assume: the VM's scratch disk is a fresh file per job, deleted after; memory is freed by the VMM's exit; the host agent runs a post-job check that the tap device, the scratch file, the per-job proxy policy and any cgroup are gone, and refuses to schedule onto a slot that fails the check. For pooled macOS guests, "destroy" means revert to the template image and rotate the guest's machine identity; a guest that cannot be reverted is retired, not reused.

#4. Workload identity instead of stored secrets

What is this?A customer's job needs to deploy to their AWS account. The old way is an access key in CI secrets, readable by any job in the repository, and by any dependency of any job. The OIDC way is that GitHub or Buildkite issues the job a signed token describing what it is, and the customer's cloud exchanges that for short-lived credentials if the claims match a trust policy.

How it works

The two mistakes to call out: a trust policy with StringLike on repo:acme/* lets any repository in the org, including a fork's pull_request workflow if the policy does not pin ref or environment, assume the production role; and a policy keyed on the repository name survives a delete-and-recreate by someone else with the same name, which is why you pin the numeric IDs. Platform-side, the job's own OIDC token is requested by the job, so the platform's job is to make sure only this job's VM can request this job's token: the runner's job token lives in the VM, the VM is single-tenant, and it is gone in minutes.

In practiceThe trust policy you want to see in a customer's account, and the one you don't:

{ "Condition": {
    "StringEquals": { "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
                      "token.actions.githubusercontent.com:repository_id": "812345",            ← immutable id, not the name
                      "token.actions.githubusercontent.com:environment": "prod" },              ← only the protected environment
    "StringLike":   { "token.actions.githubusercontent.com:sub": "repo:acme/app:environment:prod" } } }

{ "Condition": { "StringLike": { "token.actions.githubusercontent.com:sub": "repo:acme/*" } } }   ← any repo, any branch, any fork's PR workflow: do not ship this

#5. Egress: default deny, with a way to say yes

What is this?Jobs need registry.npmjs.org; they must not reach 169.254.169.254, 10.0.0.0/8, the scheduler, or a random server that a malicious dependency wants to talk to.

How it worksThree layers, each catching what the previous misses:

Why IP allowlists alone fail: registries sit behind CDNs whose addresses change hourly, so an IP list is either stale or so broad it includes the attacker's bucket on the same CDN. Why DNS-based allowlists alone fail: the job controls its own resolver and can answer "registry.npmjs.org" with any address, so the enforcement point has to be the proxy that does the lookup itself, or a datapath that pins the IP the trusted resolver returned. Why the cloud metadata service deserves its own line: a job in a cloud-burst VM that can reach IMDS gets the instance's role; the fix is both no meaningful role on the instance and a drop rule on the guest's tap, plus IMDSv2 with a hop limit of 1 so a response cannot be routed past the host.

In practiceThe proxy log is one of your best detection sources:

2026-10-24T10:02:11Z tenant=acme job=j-8f3a ALLOW CONNECT registry.npmjs.org:443   bytes_out=4.1k bytes_in=38.2M
2026-10-24T10:02:40Z tenant=acme job=j-8f3a ALLOW CONNECT github.com:443           bytes_out=2.0k bytes_in=1.1M
2026-10-24T10:03:02Z tenant=acme job=j-8f3a DENY  CONNECT 203.0.113.44:443          reason=not-allowlisted            ← a postinstall script phoning home
2026-10-24T10:03:02Z tenant=acme job=j-8f3a DENY  CONNECT 169.254.169.254:80        reason=link-local                 ← someone is probing for cloud credentials

#6. Cache partitioning and key integrity

What is this?Caches make the platform fast and are the easiest place to smuggle state from one job to another. A pull-request build that can write to a cache a release build will restore has turned a comment into code execution on the trusted branch.

How it works

cache key:    v3-npm-{{ hashFiles('package-lock.json') }}-linux-arm64
namespace:    s3://ci-cache-acme/repo-42/trusted/           ← writable only by main/release builds
              s3://ci-cache-acme/repo-42/untrusted/         ← PR builds write here, read both
manifest:     { "key": "v3-npm-9c1e…-linux-arm64", "blob": "sha256:7d20…", "writer_job": "j-77c1", "trust": "trusted", "sig": "controller-2026-10:…" }
restore:      verify sig → fetch blob by hash → verify sha256 → unpack       ← any mismatch is a miss plus an alert, never a partial restore

#7. Break-glass: debugging a job without exposing the node

What is this?A developer's job is stuck and they want a shell. On a platform the interactive path is the most dangerous feature you ship: it is a human, with a real account, inside the sandbox, often with more patience than the job had.

How it worksThe shell goes into the sandbox and nowhere else:

• An sshd starts inside the job's VM only when debugging is requested, listening on the job's private segment, reached through the proxy tier with the same per-job identity.
• Authentication is an SSH certificate minted by the controller for this user, this job and a 30-minute window, with force-command and source-address critical options, after the CI service confirmed the user has write access to the repository. No authorized_keys files on any image.
• The session user is a unique OS account created for the session, not the runner user, so file ownership attributes what the human did versus what the job did.
• Session recording is done by the controller-side proxy and, inside the VM, by an eBPF execve tracer that logs commands even if the user disables terminal echo. Logs go to the write-only sink.
• The VM's lifetime extends to the debug window and then it is destroyed like any other; there is no "keep it around" option.

Because the sshd is inside the VM, a compromised debug session has exactly the reach the job had. It cannot see the host, the co-located jobs or the control plane, because none of them are reachable from that network.

#8. Observability and provenance you can prove

What is this?Two audiences need proof: you, when something goes wrong, and customers, when they ask whether an artifact really came from their source on your platform.

How it works

In practiceThe provenance a customer can verify, abridged:

{ "predicateType": "https://slsa.dev/provenance/v1",
  "subject": [{ "name": "registry.internal/acme/app", "digest": { "sha256": "7a1f…e3" } }],
  "predicate": {
    "buildDefinition": {
      "buildType": "https://ci.example.com/buildtypes/microvm-job/v1",
      "externalParameters": { "repository": "https://github.com/acme/app", "ref": "refs/tags/v1.4.2", "workflow": ".github/workflows/release.yml" },
      "resolvedDependencies": [
        { "uri": "git+https://github.com/acme/app@refs/tags/v1.4.2", "digest": { "gitCommit": "e1b2…9f" } },   ← what the controller checked out
        { "uri": "oci://images.internal/runner-linux", "digest": { "sha256": "c3d4…10" } },                     ← golden image the job ran in
        { "uri": "https://registry.npmjs.org/", "annotations": { "fetched_via": "egress-proxy", "entries": 412 } } ] },
    "runDetails": { "builder": { "id": "https://ci.example.com/controller/v3" },                                 ← the controller, not the job
                    "metadata": { "invocationId": "j-8f3a", "startedOn": "2026-10-24T10:01:58Z" } } } }

#48–55 min: Trade-offs and failure modes

Say these unprompted; they are where senior candidates separate from mid-level ones.

🎯

On the job. The first incident on a platform like this is rarely an escape. It is a job that mines, a job that scans RFC 1918, or a token in a log. Build the proxy logs, the per-job flow summaries and the token-usage telemetry before the fancy sandboxing is finished; they catch the common case and tell you when the rare one happens.

#55–60 min: Summary and what you would build first

In practiceThe closing you want to deliver:

"Three invariants: a job can reach only the proxy and its allowlist; a job keeps nothing, because the VM, disk,
tap, token and cache credential are all per-job and destroyed together; and a job can pretend to be nothing
but itself, because registration is single-use, cloud access is OIDC pinned to immutable ids, and provenance
is signed by the controller rather than the job.

Order of build: 1) microVM-per-job with immutable hosts and the host agent's destroy check, 2) egress proxy
with default deny and logging, 3) JIT registration and the token broker, 4) partitioned signed caches,
5) break-glass via in-VM sshd with certificates, 6) controller-signed provenance.
If I could only ship one thing this quarter it would be 1 and 2 together: they bound reach and keep,
and everything else is refinement."

Then ask your question: "What does your current break-glass path look like, and who can use it?" It shows you know where the human risk lives.