Detection Engineering: From Hypothesis to Tested Alert

TL;DR: Good detections target a behaviour the attacker needs, not a tool's name. They depend on data that is actually collected everywhere it matters. They are tuned against history with narrow, owned exceptions, proven to fire by emulating the technique, and shipped with a runbook. Managed as code (versioned, tested, reviewed, staged), and measured by precision and exercise detection rate. Use ATT&CK to prioritise and find gaps; never confuse a rule's existence with coverage.

The lifecycle of one detection

1. Hypothesis. A technique worth catching, chosen from threat intelligence, choke points (credential access, privilege escalation, persistence, exfiltration) and coverage gaps.
2. Data. Which events record the behaviour, and do all relevant assets send them with the needed fields? Add health checks; a rule on missing data is silently blind.
3. Logic. Target the behaviour plus context (parent process, asset type, actor's history).
4. Tuning. Run over 30 to 90 days of history; classify matches; add narrow exceptions with owners and expiry.
5. Testing. Reproduce the technique several ways in a lab or with an emulation framework; keep the samples as test fixtures.
6. Shipping. Runbook (meaning, first checks, escalation, containment options), severity, owner, review date.
7. Operating. Precision, volume, time to triage, and whether emulation exercises still trigger it.

Precision and severity

A paging, high-severity alert needs high precision. Weak signals should not page anyone; they feed risk-based alerting, which scores entities and alerts when several signals accumulate on one host or user. This keeps recall without burning analysts out.

Detection as code

Store each rule with metadata (technique, severity, data sources, owner, runbook) and tests (positive and negative samples) in a repository. CI validates syntax, runs tests and checks metadata; reviewers check logic and false-positive risk; deployment is automated, starting in shadow mode. Scheduled end-to-end emulations confirm alerts still arrive after log-pipeline changes. Sigma is a common vendor-neutral format.

A Sigma rule (specification 2.1.0, August 2025) needs only a title, a logsource saying which kind of log it reads, and a detection block whose condition combines named selections; status (experimental, test, stable) and level (informational to critical) carry its maturity and severity. The pySigma toolchain converts rules for a specific SIEM through backends and processing pipelines that map generic field names to your schema, which is what keeps detection logic separate from vendor field names. Sigma correlation rules express counts, distinct values and ordered sequences over a time window, for example "more than 20 failed logons for distinct accounts from one source in 5 minutes", without writing each SIEM's query language by hand.

Document each detection with a fixed template so the next analyst knows what it is for. Palantir's Alerting and Detection Strategy framework, published in 2017, is the widely copied one: goal, categorisation, strategy abstract, technical context, blind spots and assumptions, false positives, validation, priority and response. The "blind spots" section is the one most teams skip and most need.

Detect the absence of data

A rule on a log source that stopped arriving is silently blind. Every source needs a health check: a heartbeat or expected event rate per host and source, and an alert when it drops. Some failures are subtle: on Windows, process-creation command lines (event 4688) are off by default and need two separate policies, and a basic audit policy can overwrite advanced audit settings, which Windows records as event 4719. Treat "this host stopped sending process events" as a detection in its own right, because attackers who disable logging rely on nobody noticing.

Coverage, honestly

ATT&CK organises tactics, techniques and procedures. Use it to choose what to build and test. Its structure changes, so coverage claims must name a version: version 18 (October 2025) replaced per-technique detection notes with first-class detection strategies and analytics, and version 19 (April 2026) split the defense evasion tactic into stealth and defense impairment, giving Enterprise ATT&CK 15 tactics. Releases come twice a year, in April and October.

Aim detections high on David Bianco's Pyramid of Pain (2013), which ranks indicators by how much it costs an attacker to change them: hash values at the bottom, then IP addresses, domain names, network and host artefacts, tools, and tactics, techniques and procedures at the top. A rule on a malware hash is trivially evaded; a rule on the behaviour the technique requires is not. A technique is covered only when tested detections or preventive controls handle the procedures you expect, on the assets that matter. One rule for one procedure does not make a technique green.

Worked example: local admin added on a server

Behaviour: an account added to a server's local Administrators group (Windows events 4720 and 4732). Data check: 7 percent of servers were not forwarding security logs; fixed first. History: 140 matches in 60 days, 131 from the provisioning service on configuration-management hosts (narrow exception), 9 from admins during approved change windows (context: change ticket). Test: four methods of adding an admin in the lab; all fire. Production: high severity on domain controllers, medium elsewhere, runbook linked, precision tracked monthly.

Put numbers on it. Before tuning, precision was 0 of 140: no historical match was malicious. After the provisioning exception and change-ticket enrichment, expected volume is about 9 alerts in 60 days, roughly one a week, each a two-minute ticket check. Recall on the tested procedures is 4 of 4, and the claim stops there: other ways of adding an administrator are not covered until they are emulated. And every rule competes for the same analysts: a team with 1,000 analyst-minutes a day for triage at 10 minutes per alert can afford about 100 alerts a day across all rules, so each rule must justify its share of that budget.

Misconceptions interviewers listen for

• "A rule mapped to a technique means the technique is covered." It covers the procedures you tested, on the assets that send the data.
• "Hash and IP indicators are good detections." They sit at the bottom of the Pyramid of Pain; attackers change them for free.
• "More alerts means more security." Volume beyond triage capacity lowers effective recall.
• "Command lines are in the Windows logs." Not by default; two policies must be enabled.
• "ATT&CK has 14 tactics, including defense evasion." Since version 19 it has 15, with stealth and defense impairment.

How this shows up in interviews

"Write a detection for X", "how do you reduce false positives", "detection as code" and "how do you measure coverage" all test this lifecycle.

Say it in 30 seconds: "I start from a behaviour the attacker needs, confirm the data that records it arrives from every asset, write logic with context, tune against history with narrow owned exceptions, and prove it fires by emulating the technique. It ships with a runbook and an owner, lives as code with tests and review, and I measure precision and exercise detection rates. ATT&CK tells me what to build and test, not what colour the map is."