Skip to content

Interview question · AI Red Teaming, Evaluation & Response

How do labs and evaluators measure an AI model's offensive cyber capability, and what do the headline numbers not tell you?

'73 percent on expert CTFs' sounds precise. A strong answer explains the task design, the attempt budget, the missing defenders and the confidence interval that sit behind a number like that.

hardNewPreparation tags: UK AI Security Institute, Anthropic, OpenAIUpdated Nov 2026

'73 percent on expert CTFs' sounds precise. A strong answer explains the task design, the attempt budget, the missing defenders and the confidence interval that sit behind a number like that.

Make your first attempt

What would you say before reading the answer?

  1. 01 State the assumptions your answer depends on.
  2. 02 Explain the mechanism and choose a control.
  3. 03 Name a condition that would change your decision.

The worked answer includes an approach, scoring criteria and follow-up questions. Compare the reasoning with your own.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 529 remaining answers · $25

Every worked answer in all 17 topics, every lesson in every course, the full concept curriculum, the complete practice-test bank, and the downloadable security field guides.

6 months · One payment · No auto-renewal

What if · the follow-ups in this answer

The answer works through 4 follow-ups like an interviewer would

A lab's announcement says its new model 'solves 73 percent of expert-level cyber challenges', and your CISO asks whether this changes your threat model.

  1. What do you need to know about the number first?
  2. What if the same model completed a multi-step range only 3 times in 10?
  3. Does this mean your network would fall to it?
  4. How should a security team use public cyber evaluations?

Each rung has a nudge, a model answer, the principle that carries over and the weak answer to avoid. Premium opens every answer and ladder.

UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

No comments yet. Share a correction or a different way to answer it.