Tabletop Exercises

I treat it as a likely leaked long-term key, because a CI user calling identity and listing APIs from an unknown hosting IP is classic post-leak reconnaissance. First I confirm in CloudTrail which access key ID was used and from where, then I contain by deactivating that access key, not deleting it, so evidence and attribution survive, and I check whether the pipeline breaks, which tells me the key really was in use. I preserve a CloudTrail export for the user from the last 30 days, open an incident channel, and page the CI owner. If the key is in a public repository, I also check GitHub's secret-scanning alerts for when it was exposed.

It means the attacker already has persistence through a new admin identity, so deactivating the original key was necessary but not enough. I immediately deactivate the backup-svc access key, attach an explicit deny-all policy to that user, then search CloudTrail for every action by both identities: other new users, roles, access keys, login profiles, trust-policy changes, new Lambda functions or EC2 instances. Anything they created gets the same treatment. I escalate severity, because the attacker had administrator rights, and bring in the incident commander and the cloud platform team.

This is now a data breach, not just an intrusion, so the work splits. On the technical side I confirm exactly which objects were read from the S3 server-access or CloudTrail data-event logs, so we can say what was taken rather than guess, and I make sure containment holds. On the business side I hand the scope to the incident commander, who pulls in legal and privacy, because regulatory clocks may have started: GDPR's 72-hour notification runs from awareness if EU personal data is involved. I don't notify customers myself; I give legal an accurate, evidenced scope so they can decide. I also rotate any secrets that could have been in those exports.

I run a blameless post-mortem focused on the system, not the developer. The immediate fixes: move CI to short-lived credentials with GitHub OIDC federation so there's no long-term key to leak, add pre-commit and server-side secret scanning to catch .env files before they're pushed, and put the customer-export bucket behind tighter access with alerting on bulk reads. I'd also ask why reconnaissance ran for fifteen minutes before anyone was paged, and whether GuardDuty's create-user and admin-attach findings should trigger automatic key deactivation. The measure of success is a specific, owned action item for each link in the chain.

I declare a ransomware incident and assume it's still spreading, because file encryption is the last step of an intrusion that has been going for days. I move the team to an out-of-band channel, since the attacker may be reading email and Teams, and I contain fast: isolate the affected hosts and the file server through EDR or by pulling network access, rather than powering them off so memory evidence survives. I disable the accounts involved and start identifying the ransomware family from the note and extension. In parallel I check whether our backups are intact and get them offline, and I call our cyber-insurer, who often must be notified before we engage anyone.

The outbound transfer means this is double extortion: they stole data before encrypting, so even a clean restore doesn't make the problem go away, and legal needs to know personal or contract data may be published. The backup being reachable from the infected network worries me, because attackers routinely delete or encrypt backups first, so I verify the snapshot is actually intact and immediately take it offline or make it immutable. I keep scoping: how the attacker got admin, which other hosts they touched, and whether domain accounts are compromised, because if they reached the domain, the whole environment is suspect.

I frame it as their decision with legal, and give honest inputs rather than a yes or no. Can we recover from backups in a timeframe the business survives: if last night's snapshot is intact and tested, that's our leverage to not pay. Is it legal: if the group or its wallet is sanctioned, paying can itself break the law, so legal must check. And what payment doesn't buy: decryptors are often slow, and paying doesn't reliably get stolen data deleted, as Change Healthcare found in 2024 when a second group extorted them with the same data. I'd also check No More Ransom for a free decryptor for this family before anyone considers paying.

Identity first, because everything trusts it: with a domain admin compromised I rebuild or clean Active Directory, reset the krbtgt account twice, reset all privileged and service accounts, and remove persistence, or the attacker just re-encrypts what I restore. Then core infrastructure: DNS, DHCP, the backup system, EDR and logging, so I can see what happens as systems come back. Then the crown-jewel business services in the order the continuity plan sets, restored from backups that predate the intrusion and patched with EDR before they touch the network. Everything else last, rebuilt from clean images where possible. And the entry point gets fixed before anyone reconnects: MFA on every remote-access path.