Skip to content
AI Security

Interview question · AI Red Teaming, Evaluation & Response

Why do you still need human review with automated judges, and how do you validate an LLM judge?

Calibrate a judge with meaningful reference labels, both error directions and correct audit weights. A high recall score alone does not explain its reported rate.

hardNewUpdated Nov 2026

Calibrate a judge with meaningful reference labels, both error directions and correct audit weights. A high recall score alone does not explain its reported rate.

Make your first attempt

What would you say before reading the answer?

  1. 01 State the assumptions your answer depends on.
  2. 02 Explain the mechanism and choose a control.
  3. 03 Name a condition that would change your decision.

The worked answer includes an approach, scoring criteria and follow-up questions. Compare the reasoning with your own.

signing in unlocks more free answers per topic

What if · the follow-ups in this answer

The answer works through 4 follow-ups like an interviewer would

A two-hundred-output audit finds twenty-four true positives, ten false positives, six false negatives and one hundred sixty true negatives against an adjudicated reference.

  1. What if someone says eighty percent recall means the reported rate is twenty percent too low?
  2. What if the audit deliberately reviews equal numbers of flagged and unflagged outputs?
  3. What if the judge cannot see the actual recipient of an operation?
  4. How should the judge enter a release assessment?

Each rung has a nudge, a model answer, the principle that carries over and the weak answer to avoid. Sign in free to open it.

UP NEXT ON YOUR JOURNEY
Next in this trackHow do you put security regression tests for prompts and guardrails into CI?Next in this trackIn an LLM red team engagement, what counts as a finding and what does not?Next in this trackHow do you assess the severity of a vulnerability in an LLM application?
DISCUSSION · 0

No comments yet. Share a correction or a different way to answer it.