SIEM Tuning, Baselining & Triage Metrics

#What Is This?

A SIEM (Security Information and Event Management) rule says "alert me when X happens." Tuning means adjusting that rule after it meets real data, so it fires on what you meant and stays quiet on everything else. Baselining means learning what normal looks like for a specific user, host, or service, so you can alert when they behave unusually rather than when they match a fixed pattern. Triage metrics are the numbers that tell you whether your rules and your analysts are healthy: how many alerts were real, how long they waited, how much analyst time each rule costs.

#Why It Matters

#The Taxonomy of Alert Noise

Every closed alert should get one of four labels. The labels decide what kind of fix is needed, so getting them right is the first tuning step.

Why the split mattersa false positive is a bug in the detection, so you fix the logic, and the fix applies everywhere. A benign true positive is a correct detection of approved activity, so you add a narrow exception for that approved context only. If you "fix" a BTP by changing the logic (for example, removing PsExec from the rule), you have deleted the detection for every attacker who uses PsExec too.

In practiceDisposition codes an analyst picks at close (the exact names vary by platform; Microsoft Sentinel uses True Positive, Benign Positive, False Positive, Undetermined):

ALERT-88412  rule=win_psexec_remote_service   host=FIN-WKS-022
  disposition: BENIGN_TRUE_POSITIVE                       ← behaviour real, approved
  reason:      CHG-20931 patch deployment by svc-patching from JMP-01
  tuning_req:  scope exception to (user=svc-patching, src=JMP-01, window=Tue 22:00-02:00)

ALERT-88419  rule=win_net_user_add            host=HELPDESK-07
  disposition: FALSE_POSITIVE                             ← behaviour did not happen
  reason:      "net user /add" string inside Outlook.exe command line (KB article link)
  tuning_req:  require Image endswith \net.exe or \net1.exe

🎯

On the job. When you review a queue, sort closures by rule and look at the free-text reasons. "Closed, known activity" with no ticket or owner is the smell: it means an analyst decided something was benign without evidence. Make the reason field mandatory and require a change ticket or asset-owner confirmation for any BTP.

#The Systematic Tuning Lifecycle

When one rule floods the queue, follow the same five steps every time. Skipping step 2 is how broad exclusions happen.

In practiceStep 2 in Splunk SPL (Search Processing Language): group a noisy rule's hits to find the dominant source.

index=notable search_name="win_encoded_powershell" earliest=-14d
| stats count by user, dest, parent_process, process_hash
| sort - count
| eventstats sum(count) as total
| eval pct=round(count/total*100,1)

user          dest        parent_process                 count   pct
SYSTEM        WKS-*       C:\Windows\CCM\CcmExec.exe      4112   86.3   ← SCCM client: approved, so BTP
jdoe          FIN-WKS-03  C:\Program Files\...\EXCEL.EXE     3    0.1   ← Excel spawning encoded PowerShell: investigate
...

The 0.1% row is the reason the rule exists. Any tuning change must leave that row alerting.

📰

Real incident — 3CX supply chain (2023, CVE-2023-29059). A trojanised version of 3CX's signed desktop phone app was pushed to customers through the vendor's normal update channel. From around 22 March 2023, endpoint products started flagging it, and users on the vendor's community forum spent about a week discussing the alerts, with many assuming a false positive and allow-listing the app. It was a North Korea–attributed supply-chain compromise. The lesson: "it's a signed vendor binary" is a reason to investigate, not a disposition. Label it BTP only after someone proves the behaviour is the expected one.

#Precision-Scoped Exclusions

An exclusion is a statement: "this exact activity, by this actor, in this place, is approved." The more fields it pins, the less an attacker can borrow it. The test for every field is: can an attacker who already has code execution on the host control this value?

Combine at least two strong fields. Never exclude a whole host, a whole user, or a whole directory.

In practiceA Sigma rule (vendor-neutral detection format) for encoded PowerShell, with a weak and a strong exclusion side by side:

title: Encoded PowerShell command line
detection:
  selection:
    Image|endswith: '\powershell.exe'
    CommandLine|contains: ' -enc'
  # BAD: attacker just names a parent CcmExec.exe in %TEMP%
  # filter_sccm:
  #   ParentImage|endswith: 'CcmExec.exe'
  filter_sccm:
    ParentImage: 'C:\Windows\CCM\CcmExec.exe'        # exact protected path, not a suffix
    User|contains: 'SYSTEM'
    ComputerName|startswith: 'WKS-'                  # workstations only; servers stay covered
  condition: selection and not filter_sccm

And the matching entry in the exception register (the list of every approved exclusion, kept in Git next to the rules):

- id: EXC-0142
  rule: win_encoded_powershell
  scope: parent=C:\Windows\CCM\CcmExec.exe, user=SYSTEM, host=WKS-*
  reason: SCCM compliance baselines run encoded scripts (CHG-20211)
  owner: endpoint-engineering@corp            ← someone to ask when it looks wrong
  approved_by: detection-lead
  expires: 2027-01-31                         ← forces a re-review, not permanent trust

📰

Real incident — Kaseya VSA / REvil (2021, CVE-2021-30116). REvil exploited the Kaseya VSA remote-management server and pushed a fake "hotfix" to managed endpoints. The payload was dropped into the agent's working folder, C:\kworking, which Kaseya's own guidance told customers to exclude from antivirus scanning. Roughly 60 managed service providers and up to 1,500 downstream businesses were hit. The lesson: a directory exclusion is an invitation, and attackers read vendor documentation too.

🎯

On the job. Admin tools are the hardest case because attackers use the same tools (PsExec, PowerShell, WMI, CI/CD runners). Scope by who and from where, not by what: PsExec is approved only for the patching service account, only from the two jump hosts, only to workstation targets. A CI/CD runner exclusion pins the runner's service account and the pipeline's job identity, and still alerts when the runner reaches out to a host or domain it has never contacted.

#Statistical and Behavioral Baselining

Fixed rules catch known patterns ("PsExec ran"). Baselines catch unusual for this entity ("this user uploaded 40 times more than their normal"). The steps:

In practiceOff-hours exfiltration baseline in SPL:

index=proxy action=allowed
| bin _time span=1d
| stats sum(bytes_out) as daily_out by user, _time
| streamstats window=30 current=f avg(daily_out) as avg_out stdev(daily_out) as sd_out by user
| eval z=round((daily_out-avg_out)/sd_out,1)
| where z>3 AND daily_out>500000000

user      _time       daily_out     avg_out     z
mkowal    2026-10-11  6.2 GB        140 MB     38.4   ← Saturday upload, 44× normal

current=f keeps today out of its own baseline; otherwise the spike inflates the average and σ it is compared against.

Where the maths breaksUpload volumes are skewed: most days are small, a few are huge. Standard deviation gets dragged around by the huge days, so teams often switch to the median and the median absolute deviation (MAD), which ignore outliers when building the baseline. New users have no history, so they need a peer-group baseline (same team, same role) until they have 30 days of their own.

UEBA (User and Entity Behaviour Analytics) is this idea productised: the platform keeps baselines per user and device across many measures and outputs an anomaly score. Treat the score as a triage input that raises priority, not as a standalone page. The identity-focused version is in identity threat detection.

📰

Real incident — Storm-0558 (2023). A China-based actor forged Microsoft authentication tokens and read the email of US government officials. The US State Department spotted it with a custom alert on MailItemsAccessed audit events: mail was being read by an application ID that did not normally access their mailboxes. Microsoft's own detections had not flagged it. The lesson: a baseline of "which apps normally touch this data" caught a stolen-key attack that no signature could.

Warning

Baselines can be poisonedAn attacker who ramps exfiltration slowly (10% more each week) teaches the baseline that the new level is normal. Defences: keep a long, fixed reference window alongside the rolling one, alert on the trend of the baseline itself, and never let one entity's baseline grow without limit.

#Correlation and Multi-Signal Detections

Many useful signals are too weak to page on alone: an encoded PowerShell command, a new scheduled task, a first-time login from a new country. Each one alone is mostly benign. Three of them on the same host within an hour almost never are.

Risk-based alerting turns weak signals into risk points attached to an entity (user or host). Nothing pages until the entity's total crosses a threshold, or until signals from several different ATT&CK tactics (attacker goals such as execution, persistence, credential access) stack up.

In practiceThe incident an analyst receives:

INCIDENT-5521  entity=host:FIN-WKS-03  risk=75  tactics=3     ← one page instead of three noisy alerts
  09:02 T1059.001  powershell.exe -enc JABz...  parent=WINWORD.EXE
  09:05 T1053.005  schtasks /create /tn "OneDriveUpdate" ...
  09:11 T1003.001  rundll32.exe → lsass.exe  GrantedAccess=0x1010

🎯

On the job. When a rule is noisy but the behaviour matters, demote it from "alert" to "risk contributor" instead of deleting it. Check that the correlation window is long enough: a patient attacker spreading steps over three days will slip under a one-hour window.

#Operational Alert Triage Workflows

Triage is the first look at an alert: is it real, how bad, who handles it. Most of it can be prepared before an analyst opens the ticket.

In practiceWhat enrichment adds to a raw alert:

raw:       user=a.nowak  src_ip=185.220.101.4  action=login_success  app=VPN
enriched:  user_role=Finance Controller     asset_criticality=high
           ip_tag=Tor exit node                                     ← changes priority instantly
           user_usual_countries=[PL]  this_login_country=DE
           prior_alerts_7d=1 (MFA push denied ×4, 08:51)            ← MFA fatigue precursor

#Triage and Detection Quality Metrics

Last verified2026-10

Metrics should answer two questions: is each rule worth its cost, and is the analyst workload sustainable?

Worked exampleA rule fires 600 times in 30 days: 3 TP, 420 BTP, 177 FP. Precision = 3 ÷ 600 = 0.5%. At 6 minutes per alert that is 60 analyst hours for 3 findings: 20 hours per true positive. Scoping the BTP exception removes 420 alerts; fixing the parser bug removes 177. The rule now fires 3 times, all real, and precision is 100%.

Vanity metrics look good on a slide and tell you nothing, or reward the wrong behaviour:

🎯

On the job. Report precision and ROI per rule, with the top ten worst rules named, every week. Pair MTTT with a quality check (a sample of closed alerts re-reviewed by a senior analyst), so speed can't be bought by sloppiness. Measure MTTD only from real incidents and purple-team exercises, where you know when the attacker started.

#Governance and Feedback Loops

Tuning decisions are security decisions, so they get the same controls as code changes.

The loop between Tier 1 analysts and detection engineers:

sequenceDiagram
    participant T1 as Tier 1 analyst
    participant Q as Case system
    participant DE as Detection engineer
    participant Git as Rules repo (CI)
    T1->>Q: Close alert<br/>disposition + reason + ticket
    Q->>DE: Weekly report<br/>worst precision / ROI rules
    DE->>Git: PR: fix logic or scoped exception<br/>+ malicious test sample
    Git->>Git: CI replays samples<br/>blocks if TP stops firing
    Git-->>T1: Release note<br/>what changed, what to watch

🎯

On the job. Give analysts a one-click "tuning request" on every alert that pre-fills rule, entity and suggested scope. If the only way to give feedback is a chat message, feedback stops and analysts quietly build their own mental exclusions instead.

#Security Angle

What attackers exploit and what to watch for: