Ephemeral SSH Access: OpenSSH Certificates, OIDC Exchange, and Kernel-Level Attribution

#Start here: the basics in plain words

Normal SSH works like a building where every door has its own list of keys taped to it. The list is the authorized_keys file on each server; your laptop's public key must be on that list before you can enter. This works for five servers. It fails for five thousand, because every hire, every leaver and every lost laptop means editing lists on every door, and nobody ever finishes removing old keys.

An SSH certificate replaces the taped-up lists with a badge system. One trusted office (the certificate authority, CA) signs a short-lived badge that says "this key belongs to alice, she may log in as deploy, and the badge expires in 15 minutes". Every server is told once: "trust badges signed by this office". After that, nobody edits the doors again. New hire? Issue a badge. Leaver? Stop issuing badges; the last one expires by itself before lunch.

The same trick works in the other direction. A host certificate is a badge the server shows you, signed by a second office (the host CA). That removes the Are you sure you want to continue connecting (yes/no)? question, which is a security control that almost everyone answers blindly.

The end-to-end picture is three parties and two signatures:

Memory hook

"badge, not key list." A key list is checked where you enter and edited by hand. A badge is issued centrally, expires on its own and carries its own rules. Every design decision on this page is a consequence of moving the decision from the door to the badge office.

#Static Keys vs OpenSSH Certificates

#What is this?

OpenSSH has supported its own certificate format since version 5.4 (2010). It is deliberately not X.509: there is no certificate chain, no ASN.1 parser, no extension explosion. A certificate is a public key with a handful of fields bolted on and one signature from a CA key. That small design is the point: the parser on the server side is tiny, and the trust model is "one flat list of CA keys", not "a forest of intermediates".

#Why it matters

Interviewers use this topic to see whether you can design access that ends by itself. Standing access is the root of most lateral movement stories: a key issued three years ago to a contractor who left, still valid on a bastion. Certificates convert "remember to revoke" into "default deny after expiry", and they move the authorisation decision to a place where you can put MFA, group lookups and logging in front of it.

#How it works

What authorized_keys cannot do, and what a certificate does instead:

In practiceA real user certificate, signed with an Ed25519 CA and inspected with ssh-keygen -L. Everything an interviewer could ask about is visible in one screen:

$ ssh-keygen -s user_ca -I 'alice@example.com oidc-sid=8f3c21' -n alice,deploy \
    -V +15m -z 4821 -O source-address=10.20.0.0/16 \
    -O no-port-forwarding -O no-agent-forwarding -O no-x11-forwarding alice.pub
$ ssh-keygen -L -f alice-cert.pub
alice-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: ED25519-CERT SHA256:j3YqN8lMKUChuFGkRRxlqesoYJQWqytgrpFGTZfqW5M
        Signing CA: ED25519 SHA256:+zFWDBrnF5v2VYVCy5/v12om26Y2UQJ4qOnYVQbfBzs (using ssh-ed25519)
        Key ID: "alice@example.com oidc-sid=8f3c21"      ← who; goes into the server log verbatim
        Serial: 4821                                      ← what the KRL revokes by
        Valid: from 2026-10-24T21:30:00 to 2026-10-24T21:46:13   ← 15 minutes, plus a minute of clock-skew slack
        Principals:
                alice                                     ← may log in as these Unix accounts
                deploy
        Critical Options:
                source-address 10.20.0.0/16               ← refused from any other source IP
        Extensions:
                permit-pty                                ← note what is missing: no agent/port/X11 forwarding
                permit-user-rc

🎯

On the job. When you inherit a fleet, the first audit is not "are keys strong" but "where do keys live and when were they last used". find / -name authorized_keys 2>/dev/null on a sample of hosts, then ssh-keygen -lf each entry and match fingerprints to people. Keys you cannot attribute are the migration's first casualties. The target state is AuthorizedKeysFile none on every host and TrustedUserCAKeys pointing at one file.

📰

Real incident — Matrix.org (2019). Attackers compromised a Jenkins server, then reached production hosts because an engineer's SSH agent was forwarded into the compromised box; the agent happily signed challenges for any host the attacker pointed it at. With certificates, the same engineer's badge would have lacked permit-agent-forwarding by default and would have expired before the pivot. The lesson: restrictions you must remember to apply per host do not scale, and agent forwarding is a loaded weapon left on the table.

#OpenSSH Certificate Internals

#What is this?

The format is defined in the OpenSSH source tree file PROTOCOL.certkeys. A certificate is an SSH public key blob whose type name ends in -cert-v01@openssh.com, followed by metadata, followed by the CA's signature over everything before it.

#Why it matters

The fields are the contract. Every question about "can a badge be misused for X" is answered by one field: can it be replayed (nonce, validity), can it be used by the wrong person (principals), from the wrong place (source-address), for the wrong thing (force-command, extensions), after compromise (serial plus KRL).

#How it works

Two rules in that layout carry most of the security weight:

In practiceThe three critical options and what each one is for:

force-command=/usr/local/bin/backup-pull     ← the only command this cert can run; the user's requested command is put in SSH_ORIGINAL_COMMAND
source-address=10.20.0.0/16,2001:db8::/32    ← refused unless the TCP connection comes from these ranges
verify-required                              ← for FIDO (sk-) keys: the hardware token must verify the user (PIN or biometric), not just presence

🎯

On the job. Review CA issuance code the way you would review an IAM policy. Red flags: -n (principals) derived from a user-supplied field without allow-listing, validity longer than the task (a day for a human is already long, an hour for a CI job is long), key IDs that are not unique per person, and extensions copied from a default that includes agent forwarding. The CA is the policy decision point; sshd only enforces what the CA wrote.

#Identity Federation and CA Architecture

#What is this?

The CA is a small service that answers one question: "given this proof of identity, which certificate may I sign?" The proof is usually an OIDC ID token from the company identity provider for humans, or a workload identity token (GitHub Actions, Buildkite, cloud instance identity) for machines. The CA verifies the token, applies policy, and signs the caller's public key.

#Why it matters

This is where the real authorisation lives, and where the interview goes deep: replay, races, token binding, and what happens when the CA key leaks. Getting the SSH part right and the OIDC part wrong gives you a very fast way to issue badges to the wrong person.

#How it works

The human flow, as a sequence:

sequenceDiagram
  participant U as Engineer CLI
  participant I as Identity provider
  participant C as SSH CA service
  participant S as Target host
  U->>I: Authorization code + PKCE<br/>(browser, MFA)
  I-->>U: Code
  U->>C: Code + verifier + ssh public key
  C->>I: Redeem code
  I-->>C: ID token (iss, sub, aud, groups, exp)
  C->>C: Verify sig, aud, nonce, jti<br/>map groups → principals
  C-->>U: Certificate, 15 min
  U->>S: SSH with cert
  S-->>U: Session, logged with key ID + serial

Design points that separate a safe CA from a demo:

In practiceThe identity-provider claims the CA actually reads, and what each becomes in the certificate:

{
  "iss": "https://login.example.com",          ← must equal the configured issuer exactly; no prefix matching
  "aud": "ssh-ca-prod",                         ← the CA's own client ID
  "sub": "00u1f2k3l4m5",                        ← stable user ID; the thing you key audit logs on
  "email": "alice@example.com",                 ← goes into the certificate key ID for humans reading logs
  "groups": ["sre-oncall", "developers"],       ← mapped through the policy table to principals and TTL
  "amr": ["mfa", "hwk"],                        ← refuse certificates for prod principals unless MFA is present
  "nonce": "sha256:4f1e…",                      ← hash of the SSH public key submitted with this login
  "exp": 1761341700,
  "jti": "a3f9c1…"                              ← stored for its lifetime; a second request with the same jti is denied
}

Where the CA private key lives decides the blast radius:

📰

Real incident — CircleCI (2023). Malware on an engineer's laptop stole a single sign-on session cookie that had already passed MFA, and the attacker used that session to reach production systems and customer secrets. For an SSH CA the same lesson applies twice: a stolen identity-provider session can request certificates, so bind issuance to device posture and fresh MFA for privileged principals, and keep TTLs short so a stolen certificate is worth minutes, not weeks.

🎯

On the job. Ask of any CA you review: can I replay a captured ID token? (Needs jti tracking.) Can I use my token to get a cert for someone else's key? (Needs key binding.) Can a token for another app be used here? (Needs strict aud.) What happens when the identity provider's signing key rotates? (Needs JWKS refresh with kid lookup, see the workload identity page.) If any answer is "yes" or "we didn't think about it", that is your finding.

#Target Host Enforcement

#What is this?

The server-side configuration that makes sshd accept certificates, decide which principals are allowed for which Unix account, present its own host certificate, and refuse revoked serials. It is four or five lines of sshd_config, each with a sharp edge.

#Why it matters

A CA that issues perfect certificates is useless if one host still accepts authorized_keys, and dangerous if the principal mapping lets alice become root. The host is where the policy is enforced, so the host is where an attacker who cannot get a certificate will look for a gap.

#How it works

# /etc/ssh/sshd_config  (relevant lines only)
TrustedUserCAKeys /etc/ssh/user_ca_keys.pub            ← one or more CA public keys; a cert signed by any of them is a candidate
AuthorizedPrincipalsFile /etc/ssh/principals/%u        ← per-account list of principal names allowed to log in as %u
RevokedKeys /etc/ssh/revoked.krl                        ← KRL; if this file is unreadable, ALL public-key auth is refused
HostKey /etc/ssh/ssh_host_ed25519_key
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub  ← served to clients alongside the host key
AuthorizedKeysFile none                                 ← the migration is not done until this line exists
CASignatureAlgorithms ssh-ed25519,rsa-sha2-512          ← refuse CAs that sign with weaker algorithms
PermitRootLogin no

Three ways to decide "which principal may become which user", from simplest to most flexible:

On the client side, host certificates need one line that replaces thousands of known_hosts entries:

# /etc/ssh/ssh_known_hosts  (fleet-wide, shipped by config management)
@cert-authority *.rack3.example.internal,10.20.0.0/16 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMq…  host-ca@infra

In practiceWhat a successful certificate login looks like in the server log. The key ID and serial are the audit trail; the fingerprint of the CA tells you which CA was trusted:

$ journalctl -u ssh -o cat | grep Accepted | tail -1
Accepted publickey for deploy from 10.20.1.54 port 51822 ssh2: ED25519-CERT SHA256:j3YqN8lMKUChuFGkRRxlqesoYJQWqytgrpFGTZfqW5M ID alice@example.com oidc-sid=8f3c21 (serial 4821) CA ED25519 SHA256:+zFWDBrnF5v2VYVCy5/v12om26Y2UQJ4qOnYVQbfBzs
                      ↑ Unix account                                                                  ↑ who (from the cert)        ↑ revoke by this   ↑ which CA

And the failure you will debug most often, which is almost always a principal or clock problem rather than a crypto one:

$ ssh -v deploy@build-07 2>&1 | grep -iE 'cert|principal|expired'
debug1: Offering public key: alice ED25519-CERT SHA256:j3Yq… agent
debug1: Server accepts key: alice ED25519-CERT SHA256:j3Yq… agent        ← sshd liked the CA signature
Permission denied (publickey).
# on the server:
error: Certificate invalid: name is not a listed principal                ← cert has alice,deploy; principals/deploy does not list either
# or:
error: Certificate invalid: expired                                       ← check `timedatectl` on both ends before anything else

macOS hosts (including Apple Silicon virtual machines started with the Virtualization framework) ship the same OpenSSH and honour the same directives; the differences are where the config lives (/etc/ssh/sshd_config.d/ is read on both) and that Remote Login must be enabled in System Settings or via systemsetup -setremotelogin on.

🎯

On the job. sudo sshd -T | grep -iE 'trusteduserca|authorizedprincipals|revokedkeys|authorizedkeysfile|hostcertificate' shows what sshd actually loaded, which is not always what the file says (a typo in an Include or a Match block silently wins). Check RevokedKeys points at a file that exists and is readable by root, because an unreadable KRL locks everyone out, including the people who could fix it.

#Bare-Metal and Multi-Arch Fleets: Killing TOFU

#What is this?

Every newly provisioned machine, whether an AMD EPYC rack server, an AmpereOne ARM node or a macOS guest on an M4 or M5 Max host, generates a fresh host key at first boot. Without host certificates, every first connection to each of them is a TOFU event, and automation that connects to hundreds of them per hour either disables host key checking or breaks.

#Why it matters

StrictHostKeyChecking no in a deployment script is one of the most common findings in infrastructure reviews, and it exists because TOFU does not fit ephemeral fleets. A host CA fixes the root cause instead of silencing the warning.

#How it works

Hard parts, honestly:

In practiceA real host certificate and the client-side check that proves TOFU is gone:

$ ssh-keygen -L -f /etc/ssh/ssh_host_ed25519_key-cert.pub
        Type: ssh-ed25519-cert-v01@openssh.com host certificate     ← type 2; useless as a user cert
        Key ID: "build-07.ampere.rack3"
        Serial: 1007
        Valid: from 2026-10-24T21:30:00 to 2027-10-23T21:31:13      ← this example is a year; production should be days
        Principals:
                build-07.ampere.rack3
                10.20.3.7                                          ← connect by IP and it still verifies

$ ssh -v deploy@build-07.ampere.rack3 2>&1 | grep -E 'host certificate|Host .* known'
debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:T174…, serial 1007 ID "build-07.ampere.rack3" CA ssh-ed25519 SHA256:uVY9… valid from 2026-10-24T21:30:00 to 2027-10-23T21:31:13
debug1: Host 'build-07.ampere.rack3' is known and matches the ED25519-CERT host certificate.   ← no prompt, no known_hosts entry for this host

📰

Real incident — GitHub RSA host key exposure (2023). GitHub's RSA SSH host private key was briefly published in a public repository, and GitHub rotated it on 24 March 2023. Every client that had TOFU-pinned the old key saw REMOTE HOST IDENTIFICATION HAS CHANGED, CI jobs using SSH checkouts failed, and millions of people ran ssh-keygen -R github.com and accepted a new key with no way to verify it beyond a blog post. Ed25519 and ECDSA users were unaffected. The lesson for your own fleet: with a host CA, a key rotation is invisible to clients, and a rotation that is not invisible is a red flag rather than a Tuesday.

🎯

On the job. Grep your automation for StrictHostKeyChecking=no, UserKnownHostsFile=/dev/null and ssh-keyscan piped into known_hosts at run time. Each one is a place where a machine-in-the-middle on the provisioning network would be accepted silently. The fix is a host CA plus one @cert-authority line in the image.

#Automated CI/CD Machine Access

#What is this?

CI jobs need SSH too: to deploy to a host, to pull from a private Git server, to fetch an artifact from a bare-metal build cache. The anti-pattern is a long-lived deploy key stored in CI secrets. The pattern is the same CA, fed by the job's workload identity token instead of a human's OIDC login.

#Why it matters

CI is where keys leak most: logs, caches, forks, compromised third-party actions. A deploy key that lives in a secrets store is one memory dump away from being on the internet (see the tj-actions incident on the workload identity page). A certificate minted per job for 10 minutes, usable only from the runner's egress IP and only for one command, is worth almost nothing to a thief.

#How it works

The claims that should decide the certificate, per platform:

In practiceThe CA's policy for a deploy pipeline, and the certificate it produces. Note that the certificate's key ID carries the run identifier, so a deploy on a host can be traced to one CI run in the server log:

# ssh-ca policy (data, reviewed like code)
- match:
    iss: https://token.actions.githubusercontent.com
    repository_id: "74213"                       # ← immutable; a renamed or re-created repo does not match
    job_workflow_ref: "acme/app/.github/workflows/deploy.yml@refs/heads/main"
    ref: refs/heads/main
  issue:
    principals: [release-bot]
    ttl: 10m
    key_id: "gha run_id={{run_id}} sha={{sha}}"
    critical_options:
      force-command: /usr/local/bin/deploy        # ← the job cannot get a shell even if the workflow is edited
      source-address: 203.0.113.0/28              # ← the runners' dedicated egress pool
    extensions: []                                # ← no pty, no forwarding of any kind

# on the target host, later
Accepted publickey for release-bot from 203.0.113.9 port 40212 ssh2: ED25519-CERT SHA256:… ID gha run_id=18223019 sha=9f1c2e7 (serial 90311) CA ED25519 SHA256:+zFW…

🎯

On the job. Search CI configuration for ssh-add, SSH_PRIVATE_KEY, id_rsa and deploy_key. Each is a long-lived credential to migrate. While migrating, at least give each existing key from= and command= restrictions in authorized_keys, so the interim state is bounded. Then measure: the number of long-lived SSH secrets in the CI secret store should trend to zero, and the CA issuance log should show one short certificate per deploy.

#Egress Filtering and Network Binding

#What is this?

source-address ties a certificate to where the connection comes from. Combined with dedicated egress IP pools (runners or bastions that always leave the network through a known, small set of addresses) and an egress proxy that only lets approved destinations through, a certificate becomes useless outside the network path it was issued for.

#Why it matters

TTL limits when a stolen badge works; source-address limits from where. An attacker who exfiltrates a certificate and its private key to their own machine has a badge that every sshd in the fleet refuses on the first packet.

#How it works

In practiceWhat binding failure looks like, which is a useful alarm because it means a certificate is being used outside the path it was issued for:

# server log
Authentication tried for deploy with valid certificate but not from a permitted source address (198.51.100.7).   ← alert on this string; it is never a normal user mistake
Connection closed by authenticating user deploy 198.51.100.7 port 55021 [preauth]

🎯

On the job. not from a permitted source address in sshd logs should page someone. Legitimate users on the VPN never trigger it; a copied certificate does. Pair it with the CA issuance log to see whose badge it was and which identity-provider session issued it.

#Session Recording and Attribution

#What is this?

Knowing who logged in is the certificate's job. Knowing what they did is the host's job, and user-space approaches (script, a recording bastion, a proxy that captures the terminal stream) only capture what the terminal shows. The reliable record comes from the kernel: every execve, every file open, every outbound connection, tagged with the session that caused it.

#Why it matters

Terminal recordings are trivially evaded and interviewers know it. The question "why is TTY logging insufficient" is really "do you understand the difference between what the user sees and what the kernel does".

#How it works

Why the terminal stream lies:

What the kernel sees instead:

Why eBPF rather than only auditd: auditd gives you the same execve facts and is the right baseline (see Linux hardening), but it is noisy at fleet scale and does not easily attach the session to each event. A small eBPF program on the sched_process_exec tracepoint, keyed by cgroup ID, produces exactly one record per command with the session attached, at a cost of microseconds. Commercial access proxies implement this pattern, and you can build a minimal version with bpftrace in a few lines.

Unique OS credentials complete the chain. If everyone logs in as deploy, the kernel's UID says deploy and attribution depends entirely on the sshd log line. If each person gets their own Unix account (created on demand from the certificate's principal by an AuthorizedPrincipalsCommand plus a PAM module, or pre-provisioned from the directory), then UID, auid, file ownership and sudo logs all say alice, and the shared role account disappears.

In practiceA minimal eBPF trace of what a session really executed, versus what the terminal recording showed. The terminal saw one obfuscated line; the kernel saw the real commands:

$ sudo bpftrace -e 'tracepoint:sched:sched_process_exec /cgroup == cgroupid("/user.slice/user-1105.slice/session-42.scope")/ { printf("%s auid=%d %s\n", strftime("%H:%M:%S", nsecs), uid, str(args->filename)); }'
21:33:02 auid=1105 /usr/bin/base64          ← the decode step
21:33:02 auid=1105 /usr/bin/sh              ← fed to a shell
21:33:02 auid=1105 /usr/bin/curl            ← what the base64 actually contained
21:33:03 auid=1105 /usr/bin/chmod
21:33:03 auid=1105 /tmp/.x/agent            ← and what it ran afterwards; none of this was visible in the TTY recording

🎯

On the job. Check grep pam_loginuid /etc/pam.d/sshd (without it, auid is unset and your audit trail starts as "unknown"), confirm auditctl -l includes -a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=unset, and look for a collector that correlates by session or cgroup rather than only by UID. If the answer to "what did serial 4821 run on build-07" takes more than one query, the design is not finished.

#Break-Glass and CA Key Management

#What is this?

The operational layer: how you revoke, how you rotate CA keys, and how you log in when the CA itself is down.

#Why it matters

Certificates move the single point of failure from thousands of files to one service. That is a good trade only if the service's failure modes are designed, not discovered.

#How it works

In practiceBuilding and testing a KRL, then revoking a serial range after a suspected intermediate compromise:

$ ssh-keygen -k -f /etc/ssh/revoked.krl -s user_ca -z 1 alice-cert.pub          ← -z is the KRL's own version number
$ ssh-keygen -Q -f /etc/ssh/revoked.krl alice-cert.pub
alice-cert.pub (alice laptop): REVOKED                                          ← serial 4821 is dead everywhere this file lands

$ ssh-keygen -k -f /etc/ssh/revoked.krl -u -s user_ca -z 2 - <<< 'serial: 4800-4900'   ← -u updates in place; one line kills 101 certificates
$ ssh-keygen -Q -f /etc/ssh/revoked.krl alice-cert.pub alice.pub
alice-cert.pub (alice laptop): REVOKED
alice.pub (alice laptop): ok                                                    ← the raw key is fine; only certificates are revoked

Warning

RevokedKeys pointing at a missing or unreadable file makes sshd refuse all public-key authentication. Ship an empty KRL in the base image before you set the directive, and monitor the file's presence like you monitor the CA.

🎯

On the job. During an incident, the revocation question is "which serials did the suspect CA or the suspect user get since time T?" That is only answerable if the issuance log records serial, key ID, principals and the identity claims. Rehearse: pick a serial, write the KRL, push it, and time how long until a login with that certificate fails on the farthest host. That number belongs in your runbook.