Build Platform Security: Running Other People's CI Jobs

#Start here: the basics in plain words

Most security pages about CI look at it from the customer's side: "how do I write a safe workflow?" This page looks from the other side. You run the build platform: the machines that run thousands of strangers' CI jobs every hour. Hosted CI services, "faster runners" vendors, remote build services and cloud dev environments all sell this.

Every job a customer sends you is arbitrary code you didn't write and can't review. It might be a normal npm test, or it might be a crypto-miner, an attempt to break out of its sandbox, or a probe for the next customer's secrets. The job must be fast, it must look like a normal Linux or macOS machine with root access, and it must not be able to touch anything outside itself.

Three ideas carry the whole page:

Memory hook

"fresh machine in, nothing out but what you meant." Each job starts from a clean image and leaves only what you explicitly keep: the log, the artifacts, the cache entry. Everything else is destroyed.

🎯

On the job: reading a company's public GitHub before an interview. A vendor's public repositories tell you what they worry about. Forks of gvisor or a virtualisation library mean sandboxing untrusted code. Their own GitHub Actions mean customers run their code with customer secrets. A "breakpoint" or SSH tool means interactive access to jobs. A cache action means cross-run state. Each maps to a section below. Ask in the interview which of these keeps them up at night.

#Isolating Untrusted Builds

What is this?Isolation is the wall between one customer's job and everything else: the host, the platform and other customers' jobs running on the same hardware.

Why it mattersCI jobs expect root access, Docker, nested containers and full network access. That makes them far more dangerous than a typical web workload. A shared-kernel container gives the job the whole Linux kernel as attack surface. One kernel bug then means every job on that host.

How it worksA job's lifecycle on a well-built platform:

Which wall to choose

The microVM's advantage is its tiny device model. Classic hypervisor escapes went through emulated hardware nobody needed. VENOM (CVE-2015-3456) was a bug in QEMU's emulated floppy disk controller. Firecracker emulates only a few devices (network, block, serial, a keyboard controller used for reset), so there is little to attack.

macOS is differentApple's licence allows at most two extra macOS virtual machines per Mac, on Apple hardware only. Cloud providers also lease Macs for at least 24 hours (that's why AWS EC2 Mac hosts have a 24-hour minimum). So macOS runners can't be packed densely like Linux ones. VMs run on Apple's Virtualization framework, and a whole physical Mac is often shared by only two jobs at a time, or dedicated to one customer. Wiping and re-imaging a Mac host is part of the security boundary.

What still crosses a VM wall

📰

Real incident — runc (CVE-2019-5736) and "Leaky Vessels" (CVE-2024-21626). In 2019, a malicious container could overwrite the host's own runc binary through /proc/self/exe and get root on the host the next time runc ran. In January 2024, runc leaked an open file descriptor to the host filesystem into new containers. A Dockerfile with WORKDIR /proc/self/fd/7 could then reach host files during an ordinary docker build. Lesson: for a platform running strangers' builds, a container escape bug is a cross-customer breach. That's why hostile CI belongs in a VM, with containers only inside the VM.

🎯

On the job. Questions to ask about any build fleet: Is there one VM per job, or are containers shared on a long-lived host? Is any VM or host reused between tenants, and how is it wiped? From inside a job, can you reach 169.254.169.254, the host's IP, or another job's IP? Run curl -m 2 http://169.254.169.254/ from a test job. It should time out.

#Docker Builds and BuildKit

What is this?BuildKit is the engine behind docker build and docker buildx. Hosted platforms often run remote BuildKit builders: a powerful machine with a warm layer cache, shared across many of a customer's jobs.

Why it mattersA Dockerfile is a program, and building it runs every RUN line. A remote builder runs many of those programs, so it is a multi-tenant sandbox too. Secrets passed to builds the wrong way get baked into image layers and shipped to everyone who pulls the image.

How it worksThe three ways a secret ends up in a build, and which one is safe:

# ✗ ARG / ENV: the value is recorded in image metadata or layers
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc && npm ci

# ✗ COPY then delete: the file still exists in the earlier layer
COPY .npmrc /root/.npmrc
RUN npm ci && rm /root/.npmrc

# ✓ Secret mount: the file exists only while this RUN step runs, never in a layer
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci

docker buildx build --secret id=npmrc,src=$HOME/.npmrc -t app .

In practiceAnyone who can pull the image can read what was baked in:

$ docker history --no-trunc registry.example.com/app:1.4
CREATED BY
RUN |1 NPM_TOKEN=npm_9fQx…aZ /bin/sh -c echo "//registry.npmjs.org/…   ← the token, in plain text
$ docker save registry.example.com/app:1.4 | tar -x && tar -tf <layer>.tar | grep npmrc
root/.npmrc                                                               ← "deleted" file, still in its layer

Security angle

📰

Real incident — BuildKit "Leaky Vessels" (CVE-2024-23651, CVE-2024-23652, CVE-2024-23653). Disclosed alongside the runc bug in January 2024. They were a race in cache mounts, a teardown step that could delete files on the host, and a privilege check that could be skipped through BuildKit's API. All three let a malicious Dockerfile or build request act outside its sandbox, and were fixed in BuildKit 0.12.5. Lesson: the build engine is part of the attack surface, not just the code being built.

#Build Caches and Remote Execution

What is this?A cache stores the output of earlier work so the next build can skip it. Examples are node_modules, compiled Go packages, Docker layers, or a whole Bazel action. A remote cache shares those outputs between machines and runs.

Why it mattersA cache entry is output someone else's build produced, which your build will use without rebuilding it. If an attacker can write one entry that a trusted build later restores, their code ends up in a trusted release. The trusted build never ran their pull request.

How it worksTwo kinds of cache, very different trust:

GitHub Actions caches are keyed caches with branch scoping:

• A run can restore caches from its own branch and from the default branch.
• A pull request's caches are scoped to that PR and are not seen by main.
• So the dangerous path is any attacker-controlled code running in a default-branch context. For example, a pull_request_target workflow that runs PR code, or an injection into a workflow on main. That code can save a poisoned cache entry, which every later run restores, including release runs.

In practiceBazel's standard setup: CI writes the remote cache, developers only read it.

# .bazelrc on developer laptops
build --remote_cache=grpcs://cache.example.com
build --remote_upload_local_results=false        ← laptops read but never write

# CI (trusted, post-merge builds only)
build --remote_cache=grpcs://cache.example.com
build --remote_upload_local_results=true

Security angle: rules for a platform that hosts caches.

📰

Real incident — Ultralytics (December 2024). A pull_request_target injection let an attacker run code in the repository's trusted context. Investigators found the attacker used that foothold to poison the GitHub Actions cache. The separate release workflow restored the poisoned cache, and the published ultralytics package contained a crypto-miner. Lesson: the release workflow had no bug of its own; the cache was the bridge.

🎯

On the job. In a design review of any cache, ask: who can write it, who reads it, and is the key chosen by the client or derived by the server? "Any job can write any key" is the finding.

#Interactive Access to CI Jobs

What is this?Some failures only happen on CI, so tools let a developer pause a job and SSH into it. Examples are a "breakpoint" step that holds the job when a test fails, or tmate-based actions that print a connection string into the log.

Why it mattersThe paused job still holds everything the job had: the checked-out code, the GITHUB_TOKEN, cloud credentials from OIDC, decrypted secrets in environment variables. An interactive shell there is a shell with all of the job's secrets.

How it works and where it goes wrong.

In practiceThe common tmate action has a setting that decides this:

- uses: mxschmitt/action-tmate@<full-sha>
  if: failure()
  with:
    limit-access-to-actor: true     # ← only the SSH keys of the user who triggered the run; without it, anyone with the log line can connect

Security angle

#Secrets and Identity on Hosted Runners

What is this?How credentials reach a job, and what the platform itself stores.

Why it mattersA CI platform holds every customer's deploy keys, cloud credentials and registry tokens in one place. That makes it one of the most valuable targets in a software supply chain.

How it works

📰

Real incident — CircleCI (January 2023). Malware on a CircleCI engineer's laptop stole a single sign-on session that was already past two-factor authentication. With it, the attacker reached production systems and took customer environment variables, tokens and keys. They were encrypted at rest, but the attacker also took the encryption keys from a running process. CircleCI told every customer to rotate every secret stored on the platform. Lesson: a CI platform concentrates its customers' secrets. Customers should prefer OIDC so there is less to steal, and the platform's staff access is part of every customer's attack surface.

🎯

On the job. For a platform: who among staff can read customer secrets, from which devices, with what session lifetime? For a customer: list every long-lived secret stored in your CI settings, and replace each one with OIDC where the target supports it.

#Shipping What You Build: Actions, CLIs and Install Scripts

What is this?CI vendors also publish software that runs inside their customers' pipelines: GitHub Actions, CLIs, SDKs, Homebrew taps, curl | sh installers.

Why it mattersCustomers run that code with their own secrets in reach. A compromised release of one popular action is a breach of every repository that uses it. It's the same pattern as the Codecov Bash Uploader (2021) and tj-actions (2025) incidents.

How to release safely

In practiceWhat a customer can check before trusting a downloaded binary:

$ gh attestation verify ./vendor-cli_linux_amd64 --repo vendor/cli
✓ Verification succeeded!
  sha256:4c1e…  was attested by:
  REPO        PREDICATE_TYPE                   WORKFLOW
  vendor/cli  https://slsa.dev/provenance/v1   .github/workflows/release.yml@refs/tags/v2.3.0   ← built by the release workflow, from this tag

🎯

On the job. Review the vendor's own release workflow like an attacker. Can a PR from a fork reach the job that publishes? Is the Homebrew tap repository protected as strictly as the main one? Does the install script download over HTTPS and check a signature, or just pipe a URL into a shell?

#Dev Environments and Preview URLs

What is this?Platforms also run cloud development environments: a browser IDE (e.g. VS Code running server-side) on a VM per developer. They also give each pull request or port a preview URL like pr-1234.acme.preview-host.dev.

Why it mattersA browser IDE is a full shell with the developer's Git credentials. A preview URL exposes an unfinished app with debug settings to the internet. Both turn a browser session into code execution.

Security angle