TL;DR: Store every detection in a repository as code plus metadata: the query or rule, the technique it targets, severity, data sources, owner, runbook link and tests. Tests include positive samples that must match (recorded or synthesised telemetry of the technique) and negative samples of known benign activity that must not. CI validates syntax, runs the tests, checks required metadata and lints for expensive queries; changes are peer-reviewed; deployment is automated, staged (shadow mode before paging) and reversible. In production, measure each detection's volume, precision and time to triage, test the logging pipeline continuously with emulated attacks, and retire or rework rules that do not earn their cost. The outcome is a detection set you can trust and change quickly.
How to approach it
Describe the repository structure and what each detection carries, then the testing strategy (the part most teams skip), then review and deployment, then operating metrics and lifecycle. Mention portability formats such as Sigma, and how to test the data pipeline as well as the rules.
A strong answer
Rule + metadata
technique, owner, runbook
Tests
positive and benign samples
Peer review
pull request
CI deploy
to the SIEM
Measure, retire
precision, value
What a detection is in the repository.
id: win-local-admin-added
title: Account added to local Administrators on a server
technique: T1136.001
severity: high
data_sources: [windows_security]
owner: detection-team-identity
runbook: runbooks/local-admin-added.md
query: |
(vendor-specific query or a Sigma rule compiled per backend)
exceptions:
- reason: provisioning service on config-mgmt hosts
owner: platform-team
expires: 2027-05-01
tests:
positive: [tests/local-admin/net-localgroup.json, tests/local-admin/powershell.json]
negative: [tests/local-admin/provisioning-run.json]
Sigma is a vendor-neutral rule format that compiles to many SIEM query languages, which helps portability and sharing; complex correlations may still need native queries.
Testing.
- Unit tests: feed recorded events (positive and negative) through the rule logic and assert matches. Captured from lab reproductions, emulation runs or sanitised incidents.
- Integration tests: run emulated techniques (adversary emulation frameworks such as Atomic Red Team or Caldera) on lab or canary hosts on a schedule, and assert that the alert appears end to end within an expected time. This catches broken log forwarding, parser changes and field renames, which silently kill detections.
- Performance checks: flag queries that scan too much data or run too long.
Review and deployment.
- Pull requests reviewed by another detection engineer for logic, false-positive risk and runbook quality.
- CI gates: syntax, tests, metadata completeness, exception expiry dates.
- Deployment pipeline pushes rules to the SIEM or data platform via API; manual console edits are disallowed or detected.
- New or significantly changed rules run in shadow mode (logged, not paged) for a period; then promoted.
- Rollback is a revert.
Operating it.
| Metric | Use |
|---|---|
| Alerts per day per rule | Spot noisy rules |
| Precision (true positives over triaged alerts) | Decide tuning or demotion |
| Mean time to triage | Runbook and context quality |
| Days since last true positive | Candidate for review or retirement |
| Integration test pass rate | Pipeline health |
| Technique coverage with tested rules | Planning |
Rules with persistently low precision are tuned, demoted to correlation inputs, or retired. Exceptions expire and are re-justified.
People and process. Analysts file feedback (false positive, missing context) that becomes issues in the repository; incidents produce new detections and tests as a post-incident action; threat intelligence drives the backlog.
AI systems. The same discipline applies to detections over AI telemetry: prompts flagged by injection classifiers, unusual tool-call sequences, agents accessing data outside their task, token-usage spikes. These need test fixtures too, including benign prompts that resemble attacks.
Console edits
Analyst changes rule live
no test
Rule silently breaks
misses attacks
Blind spot
Outcome, unsafe: No history
Noise grows
Detection as code
Rule + tests in repo
positive and benign samples
Review, CI deploy, metrics
owners, retirement
Breaks caught in CI
Outcome, safe: Changes audited
Quality measured
What interviewers probe next
- "How do you test a detection for something you've never seen in your environment?" Reproduce it in a lab with emulation tools or hand-crafted telemetry, and keep the sample as a test fixture.
- "A log source changes a field name. How would you notice?" Scheduled end-to-end emulation tests fail, and data-quality checks on expected fields alert before an attacker exploits the gap.
- "Who owns a detection?" A named team for each rule, responsible for its precision, runbook and exceptions.
What if?
What if · interview follow-ups
Hold your answer as the constraints change
A detection team manages hundreds of SIEM rules edited directly in the console, and nobody knows which ones still work.
Start here
What does detection-as-code mean?
Where does the source of truth live?
Every detection lives in a repository as code plus metadata: the query, the technique it targets, severity, data sources, owner, runbook link and tests. Changes go through review and automated deployment, like application code.
Carry it forwardDetections are versioned code with metadata.
Weak answerExporting console rules once and calling it done.
Constraint changes
What tests should each detection have?
How would you know a change broke the rule?
Positive samples that must match (recorded or synthesised telemetry of the technique) and negative samples of known benign activity that must not. Tests catch logic errors and regressions when rules or schemas change.
Carry it forwardPositive and negative samples per rule.
Weak answerTesting only that the query parses.
Constraint changes
What does CI check?
What would a reviewer want enforced automatically?
Syntax, tests, required metadata (owner, runbook, technique), and lints for expensive queries. Changes are peer-reviewed before merge. Failing checks block deployment.
Carry it forwardCI enforces syntax, tests, metadata and cost.
Weak answerRelying on reviewers to remember everything.
Constraint changes
How should deployment work?
What happens if a new rule floods the queue?
Automated, staged and reversible: new or changed rules run in shadow mode, recording matches without paging, before they alert. Rollback is a revert. Console edits are disabled or detected so the repository stays authoritative.
Carry it forwardShadow mode first, automated and reversible.
Weak answerAllowing hotfixes in the console.
Constraint changes
How do you know rules still work in production?
Would you notice a rule that silently stopped matching?
Measure each rule's volume, precision and triage time, test the logging pipeline continuously with emulated attacks, and alert when a rule that should fire stays silent. Retire or rework rules that do not earn their cost.
Carry it forwardContinuously test and measure rules in production.
Weak answerAssuming quiet rules mean no attacks.
Now defend it
How would you migrate the team?
Export rules into a repository with required metadata, add positive and negative tests starting with the highest-severity rules, enforce CI checks and peer review, deploy automatically through shadow mode with easy rollback, lock console edits, measure volume, precision and triage time, test continuously with emulated attacks, and retire rules that do not perform.
How it's scored
Weak: "Store queries in Git."
Hire: Version control with metadata, unit tests with positive and negative samples, review and automated deployment.
Strong Hire: Adds end-to-end emulation tests that catch pipeline breakage, shadow-mode rollout, performance linting, exceptions with owners and expiry, production metrics driving tuning and retirement, feedback loops from analysts and incidents, and applies it to AI telemetry.
Common mistakes
- Testing only that a query parses.
- Allowing console edits that bypass the repository.
- Never retiring rules.
- No checks that log sources still deliver the fields rules depend on.