Threat Hunting

#What Is Threat Hunting?

Detection engineering builds automated tripwires. Incident response reacts when one trips. Threat hunting is what you do in between — actively searching telemetry for malicious activity that no tripwire caught.

The defining assumption is "assume breach": instead of asking "did an alert fire?", you ask "if a competent attacker were already inside and avoiding my detections, what would they be doing, and where would the evidence be?" Then you go find out.

A hunt has two possible good outcomes:

That second point is the key: hunting and detection engineering are a loop. A successful hunt that found malicious activity by hand should almost always be automated into a detection so you never have to hunt for that same thing manually again.

#Why It Matters

#Hunting Is Hypothesis-Driven

The amateur version of hunting is "let me poke around the logs and see if anything looks weird." That doesn't scale and rarely finds anything. Real hunting starts from a specific, testable hypothesis, usually sourced from:

Bad hunt:   "Let me look at the firewall logs and see if anything's off."
Good hunt:  "Hypothesis: an attacker using DNS tunneling for C2 would generate
             abnormally long, high-entropy subdomains and high query volume to
             a single domain. Test: aggregate DNS by parent domain, flag domains
             with mean subdomain length > 30 and entropy > 4.0 bits/char."

The good hunt names the behavior, the expected evidence, and the specific query — which means it can succeed, fail, or be automated. The bad one can't.

#Hunt Methodologies

#The PEAK framework

PEAK (Prepare, Execute, Act with Knowledge) is a modern, widely referenced hunt framework. It defines three hunt types:

The PEAK loop: Prepare (pick a topic, gather data, define success) → Execute (run the hunt, gather evidence) → Act with Knowledge (document findings, create detections, hand off incidents, improve the next hunt).

#TaHiTI

TaHiTI (Targeted Hunting integrating Threat Intelligence) is a three-phase model — Initiate (turn an intel-driven trigger into an abstract hunt), Hunt (investigate across phases), Finalize (document and feed back). Its emphasis is that hunts should be driven by threat intelligence and feed results back into both intel and detection.

#The pragmatic loop (what it looks like day to day)

1. Pick a hypothesis    (from intel, ATT&CK, anomaly, or crown-jewel reasoning)
2. Identify the data     (which log source reveals the behavior? do we have it?)
3. Build the query       (aggregate / filter to surface candidate evidence)
4. Investigate results   (triage candidates: benign baseline vs. genuinely suspicious)
5. Decide:
     - found evil     → escalate to incident response
     - found a gap    → onboard the missing log source
     - found nothing  → can this be a detection? automate it.
6. Document            (hypothesis, queries, findings — so it's repeatable)

#The Pyramid of Pain in Hunting

(Covered fully in detection-engineering.md — it applies directly here.) Hunt toward the top of the pyramid. Hunting for a specific known-bad hash is barely hunting — it's a lookup, and it dies when the attacker recompiles. Hunting for behavior — "any process making outbound connections that was itself spawned by a document handler" — catches attackers regardless of their specific tools, and that's where hunting earns its cost.

#What You Hunt For (concrete examples by domain)

#Beaconing Detection: a worked hunt

C2 implants "phone home" on a schedule. Even when the traffic is encrypted, the timing leaks. A beacon produces connections at a near-regular interval (e.g., every 60s ± small jitter) with similar payload sizes — unlike human or normal app traffic, which is bursty and irregular.

Hypothesis: an implant beacons to its C2 at a fixed interval with jitter.
Hunt:
  - group network connections by (src_host, dst_ip)
  - compute the time delta between consecutive connections
  - flag pairs where the deltas have LOW variance (regular) over many connections
  - bonus signal: consistent payload size, long-lived low-volume connection
Triage:
  - exclude known-good periodic traffic (NTP, software update checks, telemetry)
  - what's left and unexplained → investigate the process and destination

This is a top-of-pyramid hunt: it doesn't care what malware family or C2 framework is used, only that it beacons — so it survives the attacker swapping tools.

#Reference — Exfiltration Channels to Hunt

Beaconing gets the attacker commands in; exfiltration gets your data out — and it's the step that turns an intrusion into a breach, so you hunt every channel it can take. There are a lot, but they share four tells, and hunting the tells beats chasing a list of domains: watch volume (more leaving than usual), destination novelty (a host/service this asset never talks to), encoding entropy (compressed/encrypted blobs, high-entropy DNS labels), and timing (off-hours or a sudden burst). Modern exfil rides trusted SaaS, so "it went to a known-good domain" is not exoneration — the channel and the destination reputation are decoupled.

Over the network (most common)

Cloud- & SaaS-native (no malware needed)

Staging & evasion (the prep you can catch before the bytes leave):

Physical & out-of-band

Memory hook

egress is egressIf bits can leave, it's a channel: DNS when egress is locked down, HTTPS-to-trusted-SaaS to blend in, archive-then-upload for bulk, and the cloud-native "share a snapshot / abuse an OAuth token" that needs no malware at all. You don't catch these with a blocklist — you catch them by volume, novel destination, entropy, and timing.

#Hunting in the Cloud

Cloud hunting is mostly API-log hunting. There are no implants to find on disk — only credentials being used in ways the real owner never would. The attacker's "process execution" is an API call, and helpfully it's all in one firehose: AWS CloudTrail, GCP Cloud Audit Logs, Azure Activity / Entra ID logs.

Here's what a single bad moment looks like — one CloudTrail record:

{ "eventName": "GetSecretValue", "eventSource": "secretsmanager.amazonaws.com",
  "sourceIPAddress": "185.220.101.42",
  "userIdentity": { "type": "AssumedRole", "arn": ".../prod-ci-runner" },
  "awsRegion": "ap-south-1" }

A CI role, reading a production secret, from a Tor exit node, in a region you've never deployed to. Four tells in four lines — and not one antivirus alert in sight. That's why you hunt the cloud in the logs.

Worked cloud hunt — snapshot exfiltrationgroup ModifySnapshotAttribute events, flag any where the added account ID is not in your org. One hit = someone is shipping a copy of a disk to an account you don't control. This is a top-of-pyramid hunt: it catches the technique (share-a-snapshot) regardless of which credential or tool the attacker used to do it.

The same hunts on GCPCloud Audit Logs are the equivalent firehose; the fields you pivot on are protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, and protoPayload.requestMetadata.callerIp. The high-signal method names:

• google.iam.admin.v1.CreateServiceAccountKey — a new long-lived key minted for a service account. Attackers love these: they don't expire and travel anywhere. Hunt every one and tie it to a change request — most workloads should use short-lived tokens and never mint a static key at all.
• SetIamPolicy that adds allUsers or allAuthenticatedUsers to a bucket or project — public exposure in a single call.
• google.iam.credentials.v1.GenerateAccessToken — service-account impersonation. A chain of these is GCP's version of role-hopping; map the chain back to the human who started it.
• AccessSecretVersion spikes from a principal or callerIp that never normally reads secrets.
• In Data Access logs, storage.objects.get / storage.objects.list enumerating a bucket of user content from an unusual principal or a datacenter IP.

The metadata pivot is identical to AWS — egress to metadata.google.internal (169.254.169.254) from something that shouldn't be asking.

#Hunting in Kubernetes

Attackers don't cd into your cluster — they kubectl exec into a pod, mount the host filesystem, or steal a service-account token and walk out with it. Kubernetes hunting splits cleanly across two telemetry layers:

The audit-log fields you'll live in: verb, objectRef.resource, objectRef.subresource, user.username, sourceIPs, responseStatus.code. Here's the cloud-native equivalent of catching an SSH session — a developer opening a shell inside a production payments pod:

{ "verb": "create",
  "objectRef": { "resource": "pods", "subresource": "exec",
                 "namespace": "prod", "name": "payments-7f9c..." },
  "user": { "username": "dev-oncall" }, "sourceIPs": ["10.4.2.9"] }

Maybe it's legitimate on-call debugging. Maybe it's an attacker with a stolen kubeconfig. Either way, you want to see every one of these.

The runtime layer is where Falco shines: a rule like "a shell was spawned inside a container" or "a process opened /etc/shadow on the node" fires on the behavior, even when the audit log looks clean because the action happened entirely inside an already-running container.

#Hunting a Consumer App at Scale

When you run a large consumer app, your scariest threats aren't always an APT in the server room — they're credential-stuffing bots hammering your login API, scrapers quietly harvesting your social graph, and the occasional insider abusing a support tool to snoop on user accounts. These hunts live in your application and identity logs, not in host EDR, and the attacker is often using perfectly valid credentials — so you hunt patterns of use, not malware.

Worked hunt — insider snoopingjoin every read in the internal admin tool to the ticketing system on the user ID being viewed, and flag any read with no contemporaneous ticket assigned to that operator. What's left is access without a business reason — exactly the pattern behind every "employee spied on users" headline. Then baseline per-operator volume: the snoop almost always views an order of magnitude more accounts than peers, so a simple peer-group outlier model catches the careful ones who do fabricate a ticket. This is a top-of-pyramid hunt — it targets the behavior (access without justification) regardless of which tool, query, or account the insider used.

#AI-Assisted Hunting

Two honest truths up front: (1) AI genuinely helps you hunt, and (2) a lot of "AI threat hunting" marketing is a WHERE anomaly_score > 0.8 wearing a confident voice. Use it where it actually earns its keep.

Where AI helps

The catch — what bites you

Rule of thumb: AI widens the funnel and writes the first draft — the human still pulls the trigger.

#Documenting Hunts

A hunt nobody recorded is a hunt you'll repeat from scratch. Each hunt should capture: the hypothesis, the data sources and exact queries used, what you found (including "nothing"), and the outcome (incident raised / detection created / gap identified). This builds an institutional library, makes hunts repeatable, and turns "we think we're covered" into "we hunted for this on these dates with these queries."