Lab: gVisor and Firecracker from zero to hero

#Level 0: words and setup

#The eight words

Everything here is about one question: when a program asks the operating system to do something, who answers?

Memory hook

three answers to "who runs the syscall?" runc: the host kernel. gVisor: a Go program pretending to be a kernel. Firecracker: a real guest kernel inside a hardware box.

#Build the lab machine

You need a Mac with Apple Silicon (M3 or newer for the Firecracker and KVM parts; M1/M2 can do Levels 1 to 6) and about 8 GB of free RAM while the lab runs. On a Linux machine with /dev/kvm, run the scripts directly and skip Lima.

Lima makes a Linux virtual machine on your Mac. Everything in this lab happens inside that VM. Your Mac is never touched.

brew install lima
cd hardening/labs/sandbox-isolation
limactl start --name sandbox-lab lima.yaml      # ~3 minutes: downloads Ubuntu, installs Docker + gVisor, downloads Firecracker, a kernel and a disk image
limactl copy -r . sandbox-lab:/tmp/lab          # put the lab files inside the VM
limactl shell sandbox-lab                       # open a shell inside the VM; you are now on Linux
cd /tmp/lab

What lima.yaml did for you, so nothing is magic:

• Booted Ubuntu 24.04 with nested virtualisation on, so this Linux VM can itself run VMs (/dev/kvm exists inside it).
• Installed Docker, then gVisor from Google's package repository, and registered it with Docker three times: runsc (default), runsc-kvm (same sandbox, using virtualisation hardware) and runsc-debug (logs every syscall it handles).
• Downloaded the Firecracker binary and jailer, plus a Linux kernel and an Ubuntu disk image from Firecracker's own CI builds, and put an SSH key into that image so you can log in to the microVM.

Check it worked:

$ uname -r; ls -l /dev/kvm; runsc --version | head -1; firecracker --version
6.8.0-142-generic                                              ← the lab VM's kernel; remember this number
crw-rw---- 1 root kvm 10, 232 Oct 15 18:04 /dev/kvm            ← virtualisation hardware is available in here
runsc version release-20261005.0
Firecracker v1.17.0

Scripts use sudo for Docker and Firecracker; that's normal inside this throwaway VM. Every script is short and commented; read it before running it.

#Level 1: what a system call looks like

Questionwhat actually happens when a program "asks the kernel"?

strace is a tool that sits between a program and the kernel and prints every request. Run it on the most boring command there is:

strace -e trace=uname uname -r

uname({sysname="Linux", nodename="lima-sandbox-lab", ...}) = 0
6.8.0-142-generic
+++ exited with 0 +++

Line by line:

• uname({sysname="Linux", ...}) = 0 — This is one syscall. The program uname called the kernel function uname(), the kernel filled in a structure (system name, host name, kernel release…), and returned 0, meaning success. The -e trace=uname flag told strace to show only this one kind of call.
• 6.8.0-142-generic — The program's normal output. It just printed the release field the kernel gave it. The program never knew the kernel version itself; it asked. Hold on to this: in Level 3 the same program will be given a different answer.
• +++ exited with 0 +++ — strace reporting the program finished.

Now count how many syscalls a tiny command needs:

strace -c ls /

------ ----------- ----------- --------- --------- ----------------
100.00    0.000991           6       150        17 total

• 150 — ls / made 150 syscalls to list one directory (open libraries, read the directory, write to the terminal…). 17 of them returned errors, which is normal: programs probe for files that may not exist.

Why this matters for the whole lab. Every one of those 150 calls is a request the kernel must parse and act on. A kernel bug is a mistake in handling one of them. A sandbox is a decision about who parses the request.

🎯

Exercise. Run strace -c -f sh -c 'echo hi | cat'. How many syscalls? Which ones create the pipe and the second process? Answer: about 187; look for pipe2 (the pipe), clone (new processes, two of them) and execve (loading cat and sh). You'll meet exactly these three again in Level 4.

#Level 2: a container is just a process

Questionwhen I docker run something, what exists on the host?

bash 00-container-is-a-process.sh

== Start a container that does nothing but sleep ==
--- the process list INSIDE the container:
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 16:37 ?        00:00:00 sleep 1000
root           7       0  0 16:37 ?        00:00:00 ps -ef
--- the same process seen from the HOST:
root       21324   21300  0 18:37 ?        00:00:00 sleep 1000

• Inside, sleep is PID 1, the first and only process, as if it were a whole machine.
• On the host, the very same sleep is PID 21324, an ordinary process with an ordinary parent. There is no machine. There's a process with a restricted view.

== The restricted view is made of namespaces ==
pid  container=pid:[4026532405]  your shell=pid:[4026531836]
mnt  container=mnt:[4026532402]  your shell=mnt:[4026531841]
net  container=net:[4026532407]  your shell=net:[4026531840]
uts  container=uts:[4026532403]  your shell=uts:[4026531838]

• Each line is one namespace, a kernel feature that gives a process its own copy of one thing: pid = process numbering, mnt = file tree, net = network interfaces, uts = hostname. The numbers are IDs. Different number = different copy. That is all a container's "isolation" is: four or five of these.

== But the kernel underneath is the same one ==
kernel seen inside:  6.8.0-142-generic
kernel on the host:  6.8.0-142-generic
a host kernel setting (ip_forward) seen inside: 1   on the host: 1

• Same kernel release. Same live kernel setting. There is no namespace for "the kernel". Every one of the container's syscalls lands in the host kernel.

== Docker can swap the program that starts containers (the runtime) ==
default runtime: runc
installed:       io.containerd.runc.v2 runc runsc runsc-debug runsc-kvm

• runc builds the namespaces above and starts the process. runsc is gVisor. Choosing one is a single flag: --runtime=runsc. The image, the command, the volumes: all unchanged.

== The same sleep, started with gVisor ==
--- inside:
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 16:37 ?        00:00:00 sleep 1000
--- host process list, looking for 'sleep 1000':
   (no 'sleep 1000' process exists on the host!)
--- what the host has instead:
21745 root  exe             runsc-gofer --log-format=json ...
21750 root  gvisor_sentry   runsc-sandbox --root=/var/run/docker/runtime-runc/moby ...

• Inside, nothing changed: sleep is still PID 1.
• On the host, there is no sleep process to find. What exists is runsc-sandbox (the Sentry, the Go kernel) and runsc-gofer (the file helper). The app lives inside the Sentry. This is the first visible difference between the two designs, and it's a big one: ps on the host can't even see what the sandboxed program is.

🎯

Exercise. Start sudo docker run -d --name t1 ubuntu:24.04 sleep 500, then find its host PID with sudo docker inspect -f '{{.State.Pid}}' t1 and run grep -E '^(Name|NSpid):' /proc/<pid>/status. What's NSpid? Answer: Name: sleep and NSpid: <hostpid> 1: the kernel records both numbers, the host's and the container's, for the one process. Clean up with sudo docker rm -f t1.

#Level 3: first contact with gVisor

Questionif I start the same image two ways, what does the program inside see?

bash 01-what-am-i-running-in.sh

== runc (the default): the container talks to the host kernel ==
kernel:   6.8.0-142-generic
version:  Linux version 6.8.0-142-generic (buildd@bos03-arm64-111) (aa
boot log: dmesg: read kernel buffer failed: Operation not permitted|
cpus:     4  mem: 5911 MB
dmi:      bios_date

• kernel — the host's real kernel, same number as Level 0.
• version — the real build string, including which Ubuntu build machine compiled it.
• boot log — dmesg reads the kernel's log. Docker's default settings forbid it from a container (Operation not permitted), but note why it had to be forbidden: the log exists, it's the host's, and it's one syscall away.
• dmi: bios_date — real firmware tables (the motherboard's vendor, BIOS date) are visible. A runc container sees the machine it's on.

== runsc (gVisor): the container talks to a kernel written in Go ==
kernel:   4.19.0-gvisor
version:  Linux version 4.19.0-gvisor #1 SMP Sun Jan 10 15:06:54 PST 2016
boot log: [    0.000000] Starting gVisor...|[    0.213018] Forking spaghetti code...|[    0.566042] Moving files to filing cabinet...|
cpus:     4  mem: 5911 MB
dmi:      product_name

• kernel: 4.19.0-gvisor — Remember Level 1: uname asks. Here the Sentry answered with a made-up, fixed version string. It's the same binary, the same uname() syscall; a different kernel replied.
• version: ... Sun Jan 10 2016 — Also fixed. gVisor doesn't pretend to be the host; it presents a stable, fake Linux.
• boot log — There was no boot, so the Sentry prints a joke log ("Forking spaghetti code"). It is allowed to be read, because it is the Sentry's own, not the host's.
• cpus / mem — gVisor passes through the real counts (it has to schedule on them).
• dmi: product_name — the Sentry serves a minimal fake /sys too. The real firmware tables are not reachable.

Try one by hand:

sudo docker run --rm --runtime=runsc ubuntu:24.04 cat /proc/cpuinfo | head -4

processor	: 0
BogoMIPS	: 48.00
Features		: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid ...
CPU implementer	: 0x61

• /proc/cpuinfo is generated by the Sentry: a plausible CPU with the real instruction-set features (programs need those to pick code paths) but no model name, no cache details, nothing that identifies the host.

What it meansUnder gVisor, a program is talking to Go code that behaves like Linux. Everything it can learn about "the kernel" is what the Sentry chooses to tell it.

🎯

Exercise. Run sudo docker run --rm --runtime=runsc ubuntu:24.04 ls /sys/class | wc -l and the same under runc. Answer: runc shows the host's full /sys/class, 69 device classes on this VM; gVisor shows 2 that it fakes. Fewer things visible = fewer things to attack.

#Level 4: who answers the syscall?

Questionwhen the program inside calls uname() 300 times, does the host kernel execute 300 uname() calls?

In Level 1 you watched strace show a syscall. Now we attach strace on the host side, to the process responsible for the container, and make the program inside do its work. If the host kernel is answering, strace will see the calls. If the Sentry is answering, it won't.

bash 02-who-answers-the-syscall.sh

== runc: trace the container's own process on the host ==
   the container's shell is host pid 14355; strace -f follows every child it starts
 52.67    0.006220          10       602           execve
 34.59    0.004085          13       301           pipe2
 12.74    0.001504           5       300           uname
100.00    0.011809           9      1203           total

• The script started a container whose shell waits 3 seconds, then runs uname -r; echo hi | cat 300 times. Strace attached to that shell (host pid 14355) with -f, "follow children", before the loop began.
• 300 uname — The host kernel executed exactly the 300 uname() calls the loop made.
• 301 pipe2 — 300 pipes from the loop plus one the shell made for itself.
• 602 execve — Each uname and each cat is a new program loaded: 2 × 300, plus the shell's own two.
• Every number matches the loop. The host kernel did the work.

== runsc: trace the Sentry (gVisor's Go kernel) and every process under it ==
   sentry is host pid 15825 (gvisor_sentry, 16 threads); its child processes: 15843 15871 15876
   (nothing: not one uname/pipe2/execve from the app reached the host kernel)
   the app really did run; ask it:
4.19.0-gvisor
  PID COMMAND
    1 sh
    4 sleep
    5 sh

• The same loop, same timing, but now strace is attached to the Sentry (all 16 threads) and to its three child processes (under the default "systrap" platform, those children are where the app's instructions actually execute).
• Result: empty. In 8 seconds of tracing, not a single uname, pipe2 or execve from the app reached the host kernel.
• The next lines prove the loop ran: the container answers 4.19.0-gvisor and its process list shows the loop's shell (PID 5) alive.

== gVisor's own syscall log shows who answered instead (runtime runsc-debug, --strace) ==
I1015 18:15:48 strace.go:563] [   2:   2] uname E uname(0xfdfffa5c1828)
I1015 18:15:48 strace.go:601] [   2:   2] uname X uname(0xfdfffa5c1828 {Sysname: Linux, Nodename: 2f5f9c2fb7af, Release: 4.19.0-gvisor, ...
I1015 18:15:48 strace.go:566] [   1:   1] sh E pipe2(0xfdffdc84f910, 0x0)
I1015 18:15:48 strace.go:604] [   1:   1] sh X pipe2(0xfdffdc84f910 [3 4], 0x0) = 0 (0x0) (7.875µs)

• This is gVisor's own strace, written by the Sentry itself (the runsc-debug runtime was installed with --strace).
• [ 2: 2] uname E uname(...) — task 2 (the uname program) Entered the uname syscall; the Sentry is logging that it received the request.
• ... X uname(... Release: 4.19.0-gvisor ...) — eXit: the Sentry answered it, filling in its fake release. Compare with Level 1's uname({sysname="Linux", ...}) = 0 from the real kernel: same call, different answerer.
• sh E pipe2 ... X pipe2(... [3 4]) = 0 (7.875µs) — the shell's pipe was created inside the Sentry in 8 microseconds, as file descriptors 3 and 4 that exist only in gVisor's world.

== A kernel feature the host has but the app should never reach: RxRPC ==
host modules containing rxrpc before: 0
--- runc:
socket(AF_RXRPC) failed: Protocol not supported
host modules containing rxrpc after:  1   ← the container made the HOST load kernel code
--- runsc:
socket(AF_RXRPC) failed: Address family not supported by protocol

• RxRPC is an obscure network protocol (it carries the AFS network filesystem). Ubuntu ships its code as a loadable kernel module, not loaded until something asks for it. It has had several memory-safety bugs over the years.
• before: 0 — the module isn't loaded.
• runc: the program asked for an AF_RXRPC socket. The request itself failed (Protocol not supported, because the exact socket type was wrong), but after: 1: the host kernel had already loaded 400 KB of RxRPC code into itself to answer. A container can drag kernel code it isn't supposed to touch into the host just by asking.
• runsc: Address family not supported by protocol, a different error from a different place. The Sentry's network stack has no RxRPC at all; it rejected the request itself. The host was never involved.

What it meansThis level is the whole gVisor idea on one screen. A kernel bug in RxRPC is reachable from any runc container on this host and simply doesn't exist for a gVisor one. Generalise: gVisor removes most of the kernel from reach, which is why Dirty Pipe fails under it on the main page.

🎯

Exercise. Do the runc half by hand. Start sudo docker run -d --name t1 ubuntu:24.04 sleep 500, get its PID as in Level 2, run sudo strace -f -e trace=uname -p <pid> in one terminal (it will wait), and in another sudo docker exec t1 uname -r. Nothing appears. Why? Answer: docker exec starts a new process whose parent is Docker's shim, not the sleep you attached to, so -f never sees it. That's the trap the script avoids by tracing the container's init before the loop starts. Attach to the exec'd process instead (sudo docker top t1 shows its host PID) and try again.

#Level 5: the guards

QuestiongVisor runs as ordinary processes on the host. What are they, and how locked down are they?

Picture the sandbox as a cell with a hatch, and the untrusted program passing notes ("open this file") through it. The guards reading the notes are the Sentry and the Gofer. An escape means fooling a guard; what the attacker wins is whatever the guard is allowed to do. So let's see what they're allowed to do.

bash 03-the-guards.sh

== Processes on the host for one gVisor container ==
    PID    PPID USER     COMMAND         COMMAND
  14379   14330 root     exe             runsc-gofer --root=/var/run/docker/runtime-runc/moby ...
  14386   14330 root     gvisor_sentry   runsc-sandbox --systemd-cgroup=true ...
  14411   14386 root     gvisor_sentry   [gvisor_sentry]
  14452   14386 root     gvisor_sentry   [gvisor_sentry]

• exe / runsc-gofer — the Gofer. (It shows as exe because gVisor re-executes itself from memory; the full command line tells you what it is.)
• gvisor_sentry / runsc-sandbox — the Sentry, the Go kernel.
• The two children of the Sentry — "stub" processes. Under the systrap platform the app's own machine code runs in these; every syscall they attempt is trapped and handed to the Sentry.
• All run as root on the host. That looks bad until the next block.

== How locked down is each guard? ==
PID      COMM           Seccomp   NoNewPrivs   Caps
14386    gvisor_sentry  2         1            000000000008001f
14379    exe            2         1            000000000004001f

• Seccomp 2 — a syscall filter is active. The Sentry may only make a few dozen kinds of host syscall; anything else is refused by the kernel before it runs.
• NoNewPrivs 1 — the process can never gain privileges again, even by running a setuid program.
• Caps — Linux splits root's power into about 40 capabilities (bits). A root shell has 1ffffffffff, all of them. 8001f is five bits. These processes are "root" in name only.

== What does each guard see as its filesystem root? ==
sentry's root contains: etc proc
gofer's root contains:  bin boot dev etc home lib media mnt opt proc root run sbin share srv sys tmp usr var

• The Sentry lives in a mount namespace (Level 2) containing etc and proc and nothing else. It cannot see the host's files, nor even the container's: it only ever receives already-opened file handles from the Gofer.
• The Gofer is chrooted to the container's root filesystem. Those directories are Ubuntu's, from the image, not the host's.

== Where do the app's writes go? ==
file written by the app            gofer?   host?    (1 = visible)
/ROOTFS-MARKER (container root)    0        0
/share/VOLUME-MARKER (volume)      1        1

• The script created two files inside the container: one at /, one in a directory mounted from the host (-v /tmp/lab-share:/share).
• The root-filesystem file never appears anywhere on the host, not even to the Gofer: gVisor keeps writes to the container's root in its own in-memory overlay. When the container ends, they're gone.
• The volume file is visible to both: an explicit volume is a deliberate hole, and it goes through the Gofer, which can reach that directory and only that directory.

== Same sandbox on the KVM platform ==
runsc-kvm sentry pid 16611 opens:  1 anon_inode:kvm-vcpu:0 ... 1 anon_inode:kvm-vm
systrap sentry pid 16425 opens:  (none)

• With --runtime=runsc-kvm the Sentry holds kvm-vm and kvm-vcpu handles: it has become a tiny guest kernel, and the CPU's virtualisation hardware now separates the app from the host as well. Two walls: Go kernel plus hardware.

🎯

Exercise. Start a gVisor container with a volume, sudo docker run -d --name v1 --runtime=runsc -v /tmp/x:/x ubuntu:24.04 sleep 500, then inside it run ln -s /etc/shadow /x/link and on the host cat /tmp/x/link. What happens and why is that fine? Answer: the host follows the symlink in the host's own /etc/shadow (permission denied unless root) — the link is just text. The danger is the reverse: a guard that follows guest-chosen paths on the host. That is exactly what the Kata virtiofsd bugs on the main page were, and the Gofer's chroot is what prevents it here.

#Level 6: where the time goes

QuestiongVisor is "slower". Slower at what?

Run this one alone; anything else running in the VM skews the numbers.

bash 04-speed.sh

runtime      5,000 small files (syscalls) hash 400 MB (CPU)
runc         358 ms                       426 ms
runsc        663 ms                       500 ms
runsc-kvm    9968 ms                      955 ms

• Column 1: create, read and delete 5,000 files: about 20,000 syscalls, almost no computation.
• Column 2: hash 400 MB of zeros: a few hundred syscalls, lots of arithmetic.
• runc — baseline. About 300 ms of each number is just container start-up.
• runsc — file churn takes ~2× longer: every one of the 20,000 syscalls is trapped and answered by the Sentry. The hash barely changes: once the work is in the CPU, gVisor isn't involved.
• runsc-kvm — looks absurd, and it's honest. This lab VM is itself a VM, so the KVM platform's traps go through two hypervisors. On bare metal, KVM is the fast platform. gVisor's documentation says exactly this: systrap on cloud VMs, KVM on bare metal. Now you've seen why rather than read it.

What it meansgVisor's cost is per syscall, not per instruction. npm install (50,000 small files) or a C build with thousands of objects is where it hurts. A model-inference loop or a video transcode isn't.

🎯

Exercise. Edit 04-speed.sh, change 5000 to 20000, and predict the three numbers in column 1 before running. Answer (measured): runc ~830 ms, runsc ~2050 ms, runsc-kvm ~167 s. The runc/runsc ratio stays ~2.5×; the per-syscall cost is constant.

#Level 7: Firecracker by hand

Questionwhat is a microVM actually made of, and how do platforms start one in milliseconds?

#7a. Talk to the API yourself

Firecracker has no "boot this kernel" command-line flag. You start the program, which does nothing but listen on a Unix socket, and then you build the machine with HTTP requests. Do it by hand once; the script in 7b does the same thing faster.

cd /opt/fc; sudo rm -f /tmp/fc.sock
sudo firecracker --api-sock /tmp/fc.sock > /tmp/console.txt 2>&1 &

Try to start it with nothing configured, just to see the API talk back:

sudo curl -s -i -X PUT --unix-socket /tmp/fc.sock -d '{"action_type":"InstanceStart"}' http://localhost/actions

HTTP/1.1 400
{"fault_message":"Start microvm error: Cannot start microvm without kernel configuration."}

• A plain HTTP API, over a socket file instead of a port. 400 with a readable message. Ask what it is:

sudo curl -s --unix-socket /tmp/fc.sock http://localhost/

{"id":"anonymous-instance","state":"Not started","vmm_version":"1.17.0","app_name":"Firecracker"}

Now build the machine, one PUT per part. Each answers HTTP/1.1 204 (success, nothing to say):

API='sudo curl -s -i -X PUT --unix-socket /tmp/fc.sock -H Content-Type:application/json'
$API -d '{"kernel_image_path":"/opt/fc/vmlinux","boot_args":"console=ttyS0 reboot=k panic=1 init=/bin/sh"}' http://localhost/boot-source
$API -d '{"drive_id":"rootfs","path_on_host":"/opt/fc/rootfs.ext4","is_root_device":true,"is_read_only":true}' http://localhost/drives/rootfs
truncate -s 64M /tmp/scratch.img
$API -d '{"drive_id":"scratch","path_on_host":"/tmp/scratch.img","is_root_device":false,"is_read_only":false}' http://localhost/drives/scratch
$API -d '{"vcpu_count":2,"mem_size_mib":128}' http://localhost/machine-config
$API -d '{"action_type":"InstanceStart"}' http://localhost/actions

• boot-source — which Linux kernel file to load, and its command line. init=/bin/sh says "when the kernel is up, run a shell instead of a whole operating system"; console=ttyS0 sends the kernel's messages to the serial port, which Firecracker connects to its own stdout (our /tmp/console.txt).
• drives/rootfs — a plain file on the host becomes the guest's disk /dev/vda. A second file becomes /dev/vdb. That's the entire "storage subsystem".
• machine-config — 2 virtual CPUs, 128 MB.
• actions InstanceStart — go.

Read the console:

grep -E 'Linux version|smp: Brought|vda\]|vdb\]|Run /bin/sh' /tmp/console.txt

[    0.000000] Linux version 6.18.54+ (root@77cfebd0834e) (gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 ...
[    0.160010] smp: Brought up 1 node, 2 CPUs
[    1.457734] virtio_blk virtio0: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)
[    1.495217] virtio_blk virtio1: [vdb] 131072 512-byte logical blocks (67.1 MB/64.0 MiB)
[    1.898072] Run /bin/sh as init process

• Linux version 6.18.54+ — a different kernel from the host's 6.8. This is a real, separate Linux, booted from the file you named. Compare Level 3, where gVisor only pretended.
• Brought up 2 CPUs — the two vCPUs you asked for.
• virtio_blk ... [vda] 1.00 GiB and [vdb] 64.0 MiB — your two files, appearing inside as disks. virtio is the standard "I know I'm in a VM, give me a simple device" protocol; Firecracker only speaks virtio.
• Run /bin/sh as init process at 1.9 s — the kernel is done and your shell is running. (Under nested virtualisation on a laptop; AWS quotes ~125 ms on bare metal.)

sudo curl -s --unix-socket /tmp/fc.sock http://localhost/

{"id":"anonymous-instance","state":"Running","vmm_version":"1.17.0","app_name":"Firecracker"}

Stop it: sudo pkill -x firecracker. The "machine" is gone; the two disk files remain.

#7b. The scripted tour

bash 05-firecracker.sh

A. Bare boot

InstanceStart → Linux booted and /bin/sh running: 2675 ms
[    2.035346] Run /bin/sh as init process

• Same as you did by hand, timed. A full Ubuntu with systemd (part B) adds ~8 s on top: Startup finished in 1.935s (kernel) + 8.743s (userspace). Platforms avoid that cost with a minimal init or, better, snapshots (part E).

B. Inside a fully booted VM (the script logs in over SSH through the tap device):

kernel:  6.18.54+   cpus: 1   mem: 231 MB
virtio devices: virtio0=0x0002 virtio1=0x0001    (0x1 = network, 0x2 = block disk)
dmi/bios tables: ls: cannot access '/sys/class/dmi/id': No such file or directory
pci devices: 0   usb: 0
RxRPC in this guest kernel: # CONFIG_AF_RXRPC is not set

• virtio devices: ... 0x0002 ... 0x0001 — the entire hardware of this machine: one disk, one network card. Compare with Level 3's runc container, which could see the host's BIOS date.
• dmi ... No such file, pci 0, usb 0 — no firmware tables, no PCI bus, no USB. Historic VM escapes went through emulated hardware nobody needed (VENOM was a floppy controller). There is nothing like that here to attack.
• CONFIG_AF_RXRPC is not set — Level 4's RxRPC trick fails here for a third reason: the guest kernel was compiled without it. runc: loaded into the host. gVisor: refused by the Sentry. Firecracker: doesn't exist.

C. The same VM from the host

USER         PID   RSS NLWP COMMAND
root       18806 98948    3 firecracker
   thread 18806  firecracker    Seccomp=2
   thread 18807  fc_api         Seccomp=2
   thread 18820  fc_vcpu 0      Seccomp=2
   open files:  23 anon_inode; 2 anon_inode:kvm-vcpu:0; ...

• One process, ~97 MB resident (the guest's own 256 MB is mapped separately), three threads: the main one, the API server, and one per virtual CPU.
• Seccomp=2 on every thread: each thread has its own filter listing only the syscalls its job needs.
• kvm-vcpu — the handle to the hardware. When the guest runs, this thread is "inside" the VM; when the guest touches a virtio device, the hardware stops it and this thread handles the request in Rust code. That hand-off is the only door between guest and host.

D. Egress: the guest's packets meet the host firewall

internet (1.1.1.1) before any rule: HTTP 000   (000 = no reply)
internet (1.1.1.1) after NAT + forward rules: HTTP 301

• The VM's network card is wired to tap0 on the host. With no rules, packets go nowhere (000: curl got no answer). Three iptables lines (NAT, forward in, forward replies back) give it the internet.

guest reads http://169.254.169.254/ : AWS_SECRET_ACCESS_KEY=platform-host-credential-DO-NOT-LEAK
after one DROP rule on tap0        : blocked
-A INPUT -d 169.254.169.254/32 -i tap0 -j DROP

• 169.254.169.254 is the address every cloud uses for the metadata service, where a machine fetches its own credentials. The script makes the host answer there with a fake secret, and the guest reads it in one curl. That is the classic CI-platform breach: a customer job steals the platform's cloud credentials.
• One rule, DROP anything from tap0 to that address, and the guest is blocked. On the main page this appears as the "On the job" check curl -m 2 http://169.254.169.254/ from inside a job: here you see both sides.

E. Snapshot and restore

snapshot written in 361 ms: mem=262144 KB  vm.state=6 KB
VMM killed.
restore API call returned in 17 ms
restore → ssh works again: 1417 ms
hello-from-before-the-snapshot
uptime: up 0 minutes

• The script wrote a file inside the VM, paused the VM, and dumped it: 256 MB of memory plus 6 KB of device state.
• It then killed the Firecracker process: the machine is gone.
• A brand-new Firecracker process loaded the snapshot in 17 ms and the VM was running again, mid-flight. SSH works ~1.4 s later (most of that is SSH's own handshake). The file is still there; uptime continued where it left off.
• No kernel boot, no systemd. Platforms keep a snapshot of a warmed-up job VM and restore a fresh copy for every job. That's how "a new VM per job" and "starts in milliseconds" are both true at once.

🎯

Exercise. In 7a, add a third drive before starting and check the console for vdc. Then try adding one after InstanceStart. Answer: the second attempt fails with PCI is not enabled: with the default MMIO transport, devices are fixed at boot. (The optional PCI transport that allows hot-plug is the one with CVE-2026-5747 on the main page. New feature, new surface.)

#Level 8: the jailer

QuestionFirecracker is the guard on the host side of the wall. What happens if a guest takes it over?

bash 06-jailer.sh

$ sudo jailer --id demo --exec-file /usr/local/bin/firecracker --uid 10000 --gid 10000 --chroot-base-dir /srv/jailer --daemonize -- --api-sock /run/firecracker.socket

• Read the flags left to right: give this VM the name demo; the program to confine is firecracker; run it as user and group 10000 (an ID with no account, no home, nothing); build its private root under /srv/jailer; detach; and everything after -- is handed to Firecracker itself.

== What the jailed VMM is, from the host ==
USER       UID     PID COMMAND
10000    10000   14217 firecracker
capabilities (CapEff): 0000000000000000
mount namespace: jailed=mnt:[4026532402]  this shell=mnt:[4026531841]
the whole filesystem it can see:
   10000        4096 dev
   10000     3377624 firecracker
   10000  1073741824 rootfs.ext4
   10000    19532288 vmlinux

• UID 10000 — not root. Compare Level 7c, where Firecracker ran as root because we started it by hand.
• CapEff: 0000000000000000 — zero capabilities. In Level 5 the Sentry still had five bits; this process has none. It cannot bind a low port, mount anything, change ownership, load a module: nothing.
• A different mount namespace (Level 2) with a four-entry filesystem: a device directory, its own binary, and the kernel and disk image copied in for this one VM. No /etc, no /home, no other tenants' VMs.

== Try to make it open a host file through its API ==
{"fault_message":"... Error manipulating the backing file: No such file or directory (os error 2) /etc/shadow"}

• We asked the API to attach /etc/shadow (the host's password hashes) as a disk. The answer isn't "permission denied"; it's "no such file". From inside the jail, that file does not exist. This is the stronger kind of restriction: not a rule that could be bypassed, but an absence.

== Boot the VM anyway and check the seccomp filter is on once it runs ==
   thread 14217  firecracker    Seccomp=2
   thread 14218  fc_api         Seccomp=2
   thread 14605  fc_vcpu 0      Seccomp=2

• Same per-thread syscall filters as Level 7c, now on top of the jail.

What it meansPut this next to the real bug on the main page. CVE-2026-5747 gave a guest potential code execution inside the Firecracker process. Under the jailer that means: code execution as user 10000, in a four-file directory, with no capabilities and a syscall filter. That is the difference between a bad day and a cross-tenant breach, and it cost one command-line wrapper.

🎯

Exercise. Run the jailer again with --uid 0 --gid 0 (and --daemonize, or it will sit in your foreground). Does it refuse? Answer: no. ps -o user,pid,args -C firecracker shows root ... /firecracker --id r2 ...: the jailer happily runs the VMM as root inside its chroot. The jailer is a tool, not a policy; --uid 10000 was a decision somebody made. When you review a platform's config, that number is one of the first things to look for, and the Kata bugs on the main page (virtiofsd as root, sandbox off) are what the wrong decision looks like.

#Level 9: applications

You now know enough to build things, not just observe them.

#9a. Write a seccomp profile that fixes the RxRPC problem for runc

Level 4 showed a runc container dragging the RxRPC module into the host. You can't switch to gVisor everywhere, but you can tell the kernel to refuse that one request. The file no-rxrpc.json is a Docker seccomp profile:

{
  "defaultAction": "SCMP_ACT_ALLOW",
  "syscalls": [
    {
      "names": ["socket"],
      "action": "SCMP_ACT_ERRNO",
      "errnoRet": 97,
      "args": [ { "index": 0, "value": 33, "op": "SCMP_CMP_EQ" } ]
    }
  ]
}

• defaultAction: ALLOW — let everything through except what's listed. (Docker's real default profile is the opposite: a long allow-list.)
• names: socket, args index 0 value 33 — when the socket() syscall is called with its first argument equal to 33 (which is AF_RXRPC)…
• action: ERRNO 97 — …don't run it; return error 97, EAFNOSUPPORT, "address family not supported". The kernel never sees a request to load the module.

sudo modprobe -r rxrpc; lsmod | grep -c ^rxrpc                                 # 0: not loaded
sudo docker run --rm --security-opt seccomp=/tmp/lab/no-rxrpc.json ubuntu:24.04 \
     perl -e 'socket(S,33,2,0) or die "socket(AF_RXRPC) failed: $!\n"'
lsmod | grep -c ^rxrpc                                                          # still 0

socket(AF_RXRPC) failed: Address family not supported by protocol
0

• Same error message gVisor gave in Level 4, now produced by the host kernel's own filter, before the socket code ran. The module stays unloaded. Normal sockets still work (getent hosts localhost succeeds inside the same container).

The lesson behind the exerciseseccomp is the cheapest wall there is: no new runtime, one JSON file. Its weakness is that you must know what to block. gVisor's advantage is that you don't: the Sentry only implements what it implements, so unknown dangerous features are absent by default.

#9b. Choose a runtime (answers hidden below each)

For each workload, pick runc, gVisor (systrap or KVM), Firecracker/Kata, or a dedicated host, and say which level's evidence backs you.

#9c. Say the blast radius in one sentence each

Practise this out loud; it's the shape of a good interview answer.

#Command cheat sheet

# runtimes
sudo docker run --rm --runtime=runsc IMAGE CMD          # gVisor
sudo docker run --rm --runtime=runsc-kvm IMAGE CMD      # gVisor on virtualisation hardware
sudo docker run --rm --security-opt seccomp=FILE ...    # custom syscall filter for runc
sudo docker inspect -f '{{.State.Pid}}' NAME            # container's host PID (runc)
pgrep -f "runsc-sandbox.*$(sudo docker inspect -f '{{.Id}}' NAME)"   # its Sentry (gVisor)
# seeing syscalls
strace -e trace=uname CMD                               # one kind, live
strace -c -f CMD                                        # count everything
sudo strace -f -c -p PID                                # attach to a running process and its future children
grep -E 'Seccomp|NoNewPrivs|CapEff' /proc/PID/status     # how locked down a process is
sudo ls /proc/PID/root/                                 # what a process sees as "/"
# firecracker
sudo firecracker --api-sock /tmp/fc.sock &              # start the VMM (it waits for the API)
sudo curl -s --unix-socket /tmp/fc.sock http://localhost/            # describe the instance
sudo curl -s -X PUT --unix-socket /tmp/fc.sock -H Content-Type:application/json -d JSON http://localhost/PATH
#   PATH = boot-source | drives/ID | network-interfaces/ID | machine-config | actions | snapshot/create | snapshot/load
sudo jailer --id ID --exec-file /usr/local/bin/firecracker --uid 10000 --gid 10000 --chroot-base-dir DIR -- FIRECRACKER_ARGS

#Clean up

exit                               # leave the VM shell
limactl stop sandbox-lab           # frees the 6 GB of RAM; `limactl start sandbox-lab` brings it back in seconds
limactl delete sandbox-lab         # removes it completely (about 5 GB on disk)

#If something goes wrong

#Which level backs which interview answer