evaluation

A patch that makes the scanner go quiet is not the same as a patch that fixes the bug. The interview is about how you would tell the difference at scale, and what the tool itself can be tricked into doing.

You can check the file and test the behaviour; you cannot prove a negative. The scored answer separates code-level threats from learned backdoors and is honest about what testing can find.

A model update can quietly undo controls you rely on. The scored answer runs a fixed security suite with thresholds agreed in advance, checks the artefact too, and decides what blocks a release.

Three definitions and one practical consequence. The consequence is the point: security tests split into deterministic checks and measured rates.

A training update can change behavior beyond its intended task. Work through a release comparison that separates useful answers, forbidden proposals and confirmed effects.

Interpret the attack and benign counts, trace a detector miss to its actual consequence, and specify the verdict timing and fallback needed for deployment.

Reproduce the failure without assuming its cause. Check meaning, labels, routing and model or guard behavior, then measure safety and useful work by language.

Trace editable source material through retrieval, actual context, model behavior and observed effects. Work through a synthetic result with denied proposals and an unresolved receipt.

A guardrail that blocks everything scores perfectly on attacks. The scored answer measures both errors, tests with attacks the guardrail has not seen, and reports with intervals at the operating threshold.

Define current task authority and forbidden effects, test policy and agent behavior together, and report proposals, outcomes and useful completion with honest denominators and uncertainty.

Separate reachable authority, forbidden proposals, gate decisions and confirmed effects. Compare designs with matched tasks, explicit denominators and uncertainty; tool counts and zero observed failures are not universal risk scores.

A defence that breaks the attacker's tools is not a defence that stops the attacker. The scored answer explains how masking fools gradient attacks, how to spot it, and how adaptive attacks get around it.

Most broken defences were evaluated by their authors. The scored answer pins down the threat model, attacks the defence specifically, runs sanity checks, and reports robust accuracy with the conditions attached.

A research-depth question. The scored answer explains why 'was this in training' blurs at web scale, why many published attacks measured distribution shift instead, and what questions remain answerable.

Generic leakage benchmarks will not tell you whether your customers' records got easier to extract. Planted canaries, matched controls and an attack-economics view of severity will.

Evasion testing is a search problem with a budget. Start from real harmful content, transform it while keeping its meaning, adapt to what gets through, and report false negatives and false positives by category.

Compare actual scope and evidence across AI red teaming, penetration testing and benchmarks. The labels alone do not tell you which boundaries were assessed.

Choose people and automation by the evidence the task needs. Validate candidates, calibrate scoring and separate exploratory search from a defensible estimate.

Calibrate a judge with meaningful reference labels, both error directions and correct audit weights. A high recall score alone does not explain its reported rate.

Turn a finding into a requirement-based test with real effect evidence. Keep unresolved runs visible and prevent reruns, tolerances or empty results from hiding a defect.

A falling aggregate can hide worse observed results in every input slice. Learn to inspect mixtures, missing outcomes, scorer changes and the release requirement.

Measure unnecessary refusal without confusing it with incomplete answers, errors or correct permission denials. Keep useful work and actual security effects visible.

Read benchmark scores through their tasks, budgets, scoring rules and missing outcomes. Work through a joint security-and-utility example before applying a result to your product.

Choose tooling from the evidence your workflow needs. Compare integration, effect scoring, state, data handling and maintenance using a representative authorized fixture.

Compare actual configurations with paired evidence, separate adaptive budgets and useful-task checks. A lower rate or significant test does not settle the release decision.

Design a bounded assessment workflow from approved requirements to verified findings. Separate genuine mechanism discovery from new wording, scorer errors and unresolved effects.

Make release a documented decision about requirements, evidence and enforceable scope. Distinguish eligible exceptions from boundaries no score or signature can waive.

Keep test purposes and exposure history explicit. Version justified corrections, split related cases and protect confirmation from both direct access and repeated tuning feedback.

Compute a zero-success bound under explicit trial assumptions, then state what it cannot establish. Keep missing effects, campaign units and release requirements visible.

Most model cards describe the model's best day. The scored answer adds what a reviewer needs: intended use, limits, evaluations with conditions, data provenance and security-relevant behaviour.

Fairness has several incompatible definitions, so the audit starts with a decision, not a metric. The scored answer picks measures for the context, gets the data to compute them, and audits the whole decision process.

Regulation does not tell you how to red-team, but it decides what you must be able to show. The scored answer maps programme outputs to the evidence each framework expects.

A composite safety score asks the reader to trust you. An assurance case states specific claims, shows the argument and evidence for each, and lists what would make it false.