Security Notes
Hands-on Labs

AWS CloudTrail + Detection Foundation Lab (Terraform)

The logging and detection half of the AWS labs. The sibling ir-scenario lab builds the attack surface (an over-privileged role, a forgotten static key, a victim EC2). This lab turns on the lights: it records every API call, alerts on the dangerous ones in near-real-time, and lets you run SQL over the history to reconstruct exactly what an attacker did.

4 min read 6 sections verified 2026-06

Last verified2026-06

CloudTrail (multi-region)
   ├─► S3 bucket            durable log archive  ──►  Athena (SQL / threat hunting)
   └─► CloudWatch Logs      ──►  metric filters ──► alarms ──► SNS  (real-time alerts)
GuardDuty                   managed threat detection (reads CloudTrail/VPC/DNS for you)

It is one flat Terraform module (not the staged 00/01/02 of ir-scenario) because nothing here depends on anything else — it's a single detection plane you stand up once.

Memory hook

The mental modelCloudWatch alarms answer "is something happening right now?"; Athena answers "show me everything that principal did last Tuesday"; GuardDuty answers "does AWS think this looks like a known attack?" You want all three, and they all feed off the one CloudTrail.

What this costs

Last verified2026-06 — verify against current AWS pricing before a long run.

ThingCost
CloudTrail management events (one trail)Free — the first copy of management events per account is free
S3 log storagePennies (logs are small; a 30-day lifecycle rule deletes them)
CloudWatch alarms~$0.10 / alarm / month × 5 alarms ≈ $0.50/mo
CloudWatch Logs ingestionPennies at lab volume (14-day retention)
GuardDutyFree for 30 days per account, then usage-based — disable after
Athena$5 / TB scanned; lab data is tiny and queries prune by region/date → pennies
S3 data events (optional)$0.10 / 100k events — off by default (enable_s3_data_events)

Comfortably inside a $180 credit. Still, terraform destroy when you're done.

Prerequisites

Same toolchain as ir-scenario (see that lab's README Step 0–1 if you haven't set up the CLI + direnv). You need:

  • A throwaway sandbox AWS account and a non-root admin CLI profile (ir-lab).
  • An S3 state bucket. Reuse the one ir-scenario/00-bootstrap already created (ir-lab-tfstate-<account-id>). If you skipped that lab, run its 00-bootstrap once — it's pennies — or point backend.hcl at any S3 bucket you own.
  • To get the most out of it, have the ir-scenario lab deployed so you have real attacker activity to detect. The detections work on their own, but the exercises in SCENARIO.md replay that lab's attacks and watch them surface here.

Spin up

bash
cd cloud/aws/labs/cloudtrail-detection
cp .envrc.example .envrc && direnv allow        # AWS_PROFILE=ir-lab, AWS_REGION=us-east-1
cp backend.hcl.example backend.hcl              # paste your state bucket name into it

terraform init -backend-config=backend.hcl
terraform apply                                 # optionally: -var 'alert_email=you@example.com'
terraform output

Useful variables (all optional):

bash
terraform apply -var 'alert_email=you@example.com'   # get the alarm emails (confirm the SNS sub)
terraform apply -var 'enable_s3_data_events=true'    # also log S3 object reads (the exfil step). Costs a little.
terraform apply -var 'enable_guardduty=false'        # if you've already used the 30-day trial

CloudTrail back-fills nothingIt only records events that happen after the trail exists. Stand this up first, then run the ir-scenario attacks — events take ~5–15 min to land in CloudWatch Logs and a bit longer in S3.

Use it

Full walkthrough — what each detection catches, how to replay the ir-scenario attacks and watch them light up, the Athena hunt queries, and the interview Q&A — is in SCENARIO.md. The short version:

bash
# Real-time: tail the audit log as you act in another shell
aws logs tail "$(terraform output -raw log_group)" --follow

# See your alarms
aws cloudwatch describe-alarms --alarm-name-prefix ir-lab- \
  --query 'MetricAlarms[].{name:AlarmName,state:StateValue}'

# History: Athena console → workgroup from `terraform output athena_workgroup`
#   → run the saved query "00 - create cloudtrail_logs table" ONCE, then the
#     "01..04" investigation queries.

Tear down

bash
terraform destroy

force_destroy = true on the buckets lets Terraform empty them first. GuardDuty's detector is deleted with everything else. Then confirm the CloudTrail, CloudWatch Alarms, and GuardDuty consoles read clean.

If you tore down ir-scenario's 00-bootstrap state bucket but it still holds THIS lab's state key, destroy this lab first. Order across labs: cloudtrail-detection → ir-scenario 02 → 01 → 00.

Notes & gotchas

One trail = free; a second trail or data events = paid.

This lab uses exactly one trail with management events. enable_s3_data_events is the only thing here that meaningfully adds cost.

Multi-region on purpose.

Attackers pick odd regions hoping you only log us-east-1. is_multi_region_trail = true captures all of them; global services (IAM, STS) log into us-east-1 regardless.

Log-file validation is on

(enable_log_file_validation). CloudTrail writes signed digest files so you can later prove the logs weren't altered — chain of custody for the S3 archive.

The bucket policy is scoped with aws:SourceArn

to this account's trail — the textbook fix for the S3 "confused deputy" (another account naming your bucket as their log destination).

Athena queries prune by region and date.

That's not cosmetic — it's what keeps "scan the whole bucket" from becoming a real (if small) bill. Widen the date filter only as far back as you need.