Performance Debugging for Slow Builds and Hanging Sessions: the USE Method, Tracing and Kernel Diagnostics

#Start here: the basics in plain words

Something is slow. A build that took four minutes now takes twelve, or an engineer says their session into a runner is "frozen". You have two jobs, in this order:

Three ideas carry most of the work:

   "it's slow"
        │
        ▼
   is it running or waiting?          ← the single most important question
   ┌────────────┴─────────────┐
 ON-CPU                     OFF-CPU
 (burning cycles)           (blocked)
   │                           │
 perf / flame graph        /proc/<pid>/wchan + stack,
 "what code?"              off-CPU trace "what's it waiting on?"
                              │
                 ┌────────────┼────────────┬───────────┐
               disk I/O     network       a lock      another process
               iostat,PSI   ss, strace    futex        parent/child

Memory hook

"running or waiting?" Ask it first, every time. The answer splits every performance problem into two halves and tells you which half of your toolbox to open. Everything else on this page is detail under that one question.

This is a security topic too, not only an operations one. Performance anomalies are a detection signal: the xz-utils backdoor (CVE-2024-3094) was caught because an engineer chased an SSH login that used suspiciously much CPU. And on a multi-tenant CI platform, "my build is slow" is often "another tenant is saturating a shared resource", which is a tenancy and fairness question as much as a speed one.

#Why it matters

Interviewers use performance debugging to tell whether you treat the host as a black box or understand what the kernel is doing. "I'd add more CPU" is a black-box answer. "I'd check whether it's on-CPU or off-CPU, read wchan to see which kernel function it's blocked in, and confirm with PSI before touching anything" is a systems answer.

It also maps directly onto hardened infrastructure. In a world of gVisor sandboxes, cgroup-limited runners and egress proxies, the slow-down is frequently caused by the security layer: syscalls reflected through gVisor's user-space kernel, a cgroup CPU cap throttling the job, a TLS handshake hanging on a proxy. If you cannot tell "the sandbox is doing its job and it costs 2x" apart from "something is broken", you cannot run the platform.

#The USE method: a checklist so you don't miss a resource

Brendan Gregg's USE method is a way to not forget anything. For every resource, check three things:

Walk the resources: CPU, memory, disk I/O, network. For each, you have a one-line command for U, S and E. The point of the checklist is coverage: people reliably find CPU and forget disk, or find utilization and forget the error counter that was the whole story.

RED vs USEYou will be asked the difference. USE is resource-centric, for one machine: is any resource exhausted? RED is service-centric, for a request-handling service: Rate (requests/sec), Errors (failed requests), Duration (latency). Use USE to debug a slow host; use RED to watch a service. A slow CI runner is a USE problem; a slow API behind it is a RED problem.

In practiceThe first-60-seconds reconnaissance on the lab VM. Each line answers one USE cell:

$ uptime
 load average: 0.32, 0.29, 0.28      ← 4 CPUs, load < 1: not CPU-saturated

$ vmstat 1 3
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 138460  98648 4847524   0    0    14   175 3476   12  1  3 96  0  0
 1  0      0 138460  98648 4847612   0    0     0    12 3529 4725  0  1 98  0  0
#  ^r=run queue  ^si/so=swap (0 = not memory-saturated)        ^wa=0 (not disk-blocked)  ^id=98 idle

$ for f in cpu memory io; do echo "$f: $(cat /proc/pressure/$f | head -1)"; done
cpu: some avg10=1.53 avg60=1.81 avg300=2.09       ← ~2% of the time *something* waited for CPU
memory: some avg10=0.00                            ← nothing ever waited for memory
io: some avg10=0.00                                ← nothing waited for disk in the last 10s

This machine is healthy: idle CPU, no swap, no I/O wait, flat pressure. That negative result is worth as much as a positive one, it rules out three whole resources in four commands, so you can stop looking at the host and look at the application.

🎯

On the job. Pressure Stall Information (PSI), the /proc/pressure/* files, is the metric to graph per CI runner. some avg10 rising on cpu, memory or io tells you which resource is the bottleneck and how long work is stalled on it, in one number, without a sampling tool. A platform that alerts on PSI catches "runners are thrashing" before engineers file "CI is slow" tickets.

#Off-CPU: finding what a frozen process is waiting on, without touching it

When a process is stuck, the most useful facts are already in /proc, free and non-invasive. Two files answer "what is it blocked on?":

In practiceA pipeline where one command waits for input that never comes, the shape of a hung build step waiting on a child:

$ (sleep 60 | cat) &                      # cat blocks reading a pipe the sleep hasn't written
$ grep State /proc/$(pgrep -n cat)/status
State:  S (sleeping)                        ← interruptible sleep: waiting, not working
$ cat /proc/$(pgrep -n cat)/wchan
pipe_read                                   ← one word: it is blocked reading a pipe
$ sudo cat /proc/$(pgrep -n cat)/stack
[<0>] pipe_read+0x40c/0x480                 ← confirmed: parked inside the kernel's pipe_read
[<0>] vfs_read+0x318/0x348
[<0>] ksys_read+0x118/0x140
[<0>] __arm64_sys_read+0x28/0x50
$ ls -l /proc/$(pgrep -n cat)/fd
0 -> pipe:[3149574]                         ← stdin is a pipe; nobody is writing the other end

In four reads, and without stopping the process, you know it is not slow but waiting, waiting on a pipe, for a writer that is itself asleep in a timer. No profiler needed. The same reads diagnose a frozen SSH session: wchan of do_select or n_tty_read means it is waiting for terminal input or a poll, not deadlocked.

Reading the sleep stateThe State letter matters:

Off-CPU over timewchan is a snapshot. To learn where a process spends its blocked time across a whole slow run, an eBPF off-CPU trace sums the time each thread spends switched out:

$ sudo bpftrace -e 'tracepoint:sched:sched_switch {...}'   # off-CPU time per command, abridged
@offcpu_ms[dockerd]: 8905          ← dockerd spent ~8.9 s of the window blocked (mostly idle waiting for work)
@offcpu_ms[yes]: 0                 ← the CPU hog never blocks: its problem, if any, is on-CPU

A build step that shows large off-CPU time is waiting, not computing; one that shows near-zero off-CPU time while running slowly is CPU-bound and wants a profiler.

📰

Real incident — the xz-utils backdoor, CVE-2024-3094 (2024). A maintainer hid a backdoor in the liblzma compression library that, when linked into OpenSSH's server through a distribution patch, let an attacker with the right key run commands on the host. It was caught not by a scanner but by performance debugging: an engineer noticed SSH logins were using far more CPU than they should and that a profiling run under Valgrind threw errors, pulled the thread, and found the backdoor before the affected versions reached stable releases. Lesson: a login that is suddenly slower, a process burning CPU it has no reason to, is a security signal, not just an annoyance. The same off-CPU/on-CPU instincts that debug a slow build are what caught a supply-chain implant.

#On-CPU: profiling code that is genuinely busy

If the process is R and slow, it is executing too much code, and you want to know which code. perf samples the running stack many times a second and tells you where the time concentrated.

The frame-pointer problemA profiler reconstructs "which function called which" by walking stack frames. Compilers that omit frame pointers for speed (common for Go, and for C/Rust built -O2 without -fno-omit-frame-pointer) leave the walker guessing, and you get a flat list of raw addresses instead of a readable tree. The fix is to build the hot code with frame pointers, or use DWARF/last-branch-record unwinding. On the lab VM, perf without frame pointers shows exactly that degraded output:

$ sudo perf record -F 99 -g -- gzip -1 -c big.bin >/dev/null
$ sudo perf report --stdio --no-children --percent-limit 3
    18.37%  gzip  gzip  [.] 0x00000000000030e4     ← raw address, not a function name:
            ---0xc9e583aa30e4                        no frame pointers / symbols in this build
    15.31%  gzip  gzip  [.] 0x000000000000cf90

A flame graph turns those samples into a picture: the x-axis is the share of samples (so a wide box is where time went, width is what matters, left-to-right order does not), the y-axis is call depth. You read it by scanning for the widest boxes near the top: those are the functions actually on-CPU. It is the fastest way to answer "where is the build spending its compile time?" across thousands of short-lived compiler processes.

PermissionsProfiling reads hardware counters, which is itself gated:

$ cat /proc/sys/kernel/perf_event_paranoid
4                                   ← >=2 disallows kernel profiling; perf needs CAP_PERFMON or a lower value

That gate is a security control: unrestricted performance counters are a side-channel risk, so hardened hosts raise perf_event_paranoid. On a locked-down runner you profile with elevated privilege in a controlled window, not by weakening the host.

🎯

On the job. When a CI owner asks "why did compile time double?", a flame graph of one representative build usually shows it in one glance: a dependency that regressed into a hot loop, a debug build flag left on, a linker doing far more work than before. Capture it on one slow build rather than guessing from averages.

#Dynamic tracing with strace: precise, and dangerous to the subject

strace prints every syscall a process makes, with arguments, return value and timing. It is unbeatable for "what file is it trying to open?" or "where does the network call stall?". It is also the tool most likely to turn a slow process into a stuck one, because it works by stopping the target on every single syscall and handing control to the tracer.

In practiceFirst, the good use, timing a network call to find the stall (-tt timestamps, -T per-call duration):

$ strace -f -tt -T -e trace=connect curl -s -o /dev/null https://example.com
21:32:35.435907 connect(3, {AF_UNIX, "/var/run/nscd/socket"}) = -1 ENOENT <0.000081>   ← name cache miss, instant
21:32:35.444294 connect(7, {AF_INET, 53, 127.0.0.53})        = 0 <0.000066>             ← DNS to local resolver
21:32:35.467847 connect(7, {AF_INET, 443, 104.20.23.154})    = 0 <0.000078>             ← TLS target; gap before it = DNS time

The timestamps localise the delay: time between the DNS connect and the port-443 connect is DNS resolution; a long <...> on the 443 connect itself is a network or firewall stall. This is how you tell "DNS is slow" from "the egress proxy is dropping us" without guessing.

Now the danger, measured on the lab VM, a tight loop of 20,000 syscalls run three ways:

plain:                         0.12 s
under strace -f:              27.47 s      ← ~230x slower: every syscall round-trips through the tracer
under strace -e trace=connect: 26.35 s     ← filtering the *output* doesn't help; it still stops on every syscall

The filter reduces what is printed, not what is intercepted, so it barely helps. On a high-throughput multithreaded Go or TypeScript build, that slowdown can trip timeouts, change the timing enough to hide the bug (a heisenbug), or cause the very hang you are chasing.

Lower-overhead alternatives

📰

Real incident — strace-induced deadlock pattern (ongoing). Attaching strace -p to one thread of a running multithreaded daemon is a well-known way to deadlock it: strace stops the traced thread on a syscall while other threads keep running and block on a lock the stopped thread holds. There is no single CVE; it is a recurring outage cause. Lesson: strace -p on a production, multithreaded, lock-heavy process is a change to the system, not a passive observation. On anything you can't afford to perturb, read /proc first and use eBPF for live tracing.

#The sandbox tax: diagnosing gVisor and microVM overhead

On hardened runners the "slowness" is often the isolation layer doing its job, and the skill is telling expected cost apart from a real bottleneck.

gVisor (runsc)gVisor answers the workload's syscalls in a user-space kernel (the Sentry) and routes file operations through a host helper (the Gofer). Both add cost per operation, so CPU-bound code runs near native while small-file, syscall-heavy work (an npm ci, a BuildKit build compiling thousands of files) slows noticeably. The lab VM shows the shape, a 3,000-tiny-file workload under each runtime:

$ time docker run --runtime=runc  alpine sh -c 'for i in $(seq 3000); do echo x >/tmp/f$i; done'
runc:   0.34 s
$ time docker run --runtime=runsc alpine sh -c '...same...'
runsc:  0.50 s                      ← ~1.5x here; on real builds with large files the gap widens toward 2x

To decide where the gVisor cost lands, syscall emulation (Sentry) or file I/O (Gofer), turn on its strace log and look at the per-call durations:

$ docker run --runtime=runsc-debug alpine sh -c 'cat /etc/hostname'   # --strace enabled
cat E openat(AT_FDCWD, "/etc/hostname", O_RDONLY) ...
cat X openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3 (278.879µs)     ← one open took 279µs through the Gofer

If opens and reads dominate (hundreds of µs each, many of them), the bottleneck is file-system reflection through the Gofer, and the fix is fewer files / a cache / a different storage mode, not more CPU. If compute dominates the flame graph and syscalls are cheap, gVisor is not your problem. The sandbox isolation deep dive covers the platform modes that change this.

microVMs and Apple Virtualization (vz)Inside a Firecracker/Kata microVM or an Apple vz guest on Apple Silicon, the guest has its own kernel, so wchan, PSI and iostat all work normally inside the guest, that is where you debug. The extra question is the virtual device: a hang on virtio-fs (file sharing into the guest) or a virtio-net stall looks like disk or network trouble inside the guest but is really the host-side device daemon. On vz you cannot perf the host kernel from inside the guest, so you profile in the guest and correlate with host-side CPU separately. The decision between the two sandbox families is in the sandbox isolation deep dive.

🎯

On the job. When someone reports "builds on the sandboxed runners are 4x slower", the first move is not to remove the sandbox. It's to establish the shape: CPU-bound (flame graph) or syscall/IO-bound (strace-log durations, PSI io). gVisor's tax is on the second shape, and the answer is usually caching the dependency fetch, not abandoning isolation.

#Network stalls: the slowness that looks like a hang

A session or download that "freezes" is often a network wait, and the socket state names the cause. ss -tanpi and /proc/net/tcp expose it.

$ ss -tanpi dst :443
State       Recv-Q Send-Q  Peer                 Process
CLOSE-WAIT  415    0       104.20.23.154:443    python3 pid=101449     ← app never close()d after peer hung up
            cubic rtt:12.9/7.6 cwnd:10 ...                             ← healthy RTT; the stall is the app, not the net

Reading the states:

TLS handshake hangs (common behind an egress proxy or an automatic-certificate service) show as a connected socket with no application data flowing, the TCP connect succeeded but the handshake stalled, which you see as a long gap after connect in an strace and a live-but-silent socket in ss. SSH keepalive failures show as a session that dies silently through a stateful firewall or NAT that dropped the idle flow; ClientAliveInterval on the server keeps the flow warm and surfaces a dead peer promptly instead of hanging. For the TCP header fields and congestion-control mechanics behind these, see the tcpdump & eBPF deep dive.

🎯

On the job. "Downloads from the package registry hang intermittently on the runners" is, four times out of five, a SYN-SENT stuck against the egress proxy or a dropped idle TLS flow, not a slow mirror. ss -tanpi from inside the job during a hang tells you which, and whether the proxy allowlist is quietly dropping a host the build newly depends on.

#CPU saturation vs cgroup throttling

A build can be CPU-bound two ways that look identical in top and need opposite fixes. True saturation is more runnable work than cores, visible as a run queue (r in vmstat) above the core count and a rising CPU PSI. cgroup throttling is the job hitting a cpu.max ceiling the platform set, visible only inside the cgroup, the CPU may be mostly idle at the host level while the job crawls.

In practiceA container capped at half a core, running two busy loops:

$ docker run --cpus=0.5 alpine sh -c 'yes>/dev/null & yes>/dev/null & sleep 3; cat /sys/fs/cgroup/cpu.stat'
nr_periods 31
nr_throttled 31              ← throttled in every scheduling period
throttled_usec 4552700       ← 4.5 s parked waiting for quota, while host CPUs sat ~idle

The host's vmstat would show idle CPU the whole time; only cpu.stat inside the cgroup reveals the job is capped. If you debug this from the host you conclude "CPU is fine" and never find it. The lesson for multi-tenant platforms: always read the job's own cgroup, not just the host.

📰

Real incident — the Cloudflare regex outage (2 July 2019). A single firewall rule with a regular expression that backtracked catastrophically drove CPU to 100% on every machine handling traffic, taking the service down worldwide for about 27 minutes. It is the canonical "on-CPU saturation" story: not a leak, not I/O, just one piece of code burning every available cycle, and a CPU flame graph pointed straight at the regex engine. Lesson: runaway CPU is often one hot function, and a flame graph finds it in one capture; and a resource limit (a CPU budget, a regex step limit) turns "the fleet is down" into "one rule is slow".

#A diagnostic order that doesn't waste time

Put it together as a sequence, cheapest and least invasive first:

Tip

See it yourselfThe sandbox-isolation lab Level 6 ("where the time goes") runs the same workload under runc and runsc and has you read the per-syscall cost, the exact gVisor-tax measurement this page describes.