What AI Red Teaming Is

TL;DR: AI red teaming is authorized adversarial assessment against defined security or safety objectives. Its value is the evidence it produces about a model or application under stated conditions, including what a realistic actor can influence and what harm follows. Select exploratory, repeatable, manual and automated methods by the question being asked; none of those labels substitutes for scope, sound measurement or a useful finding.

What you should be able to do

Turn a product concern into an assessment objective, select methods that can answer it and distinguish a demonstrated boundary failure from a signal or an unknown effect. Write a short finding with supported impact, reproduce the relevant conditions safely, and explain what the assessment cannot establish about the release.

Start with a requirement someone can check

In a fictional service, Kira reviews a customer-support assistant that explains orders and prepares draft replies. Customers may see their own order details. Staff can prepare internal notes, but customers must not receive those notes through replies, drafts or shared transcripts. The assistant should also answer legitimate delivery questions accurately.

A useful objective is to assess whether customer-controlled messages or ticket content can cause a prohibited release of an internal note. Define the actor's access, the protected record, the recipient and the relevant service rule. “Find a jailbreak” omits the consequence; “make every answer safe” is too broad to measure.

The authorized assessment uses isolated test tenants, synthetic records and controlled effect receivers. No real customer information or live external messages are needed to investigate this boundary. Test authority is its own grant: discovering a reachable integration does not authorize assessment of that integration.

The object can be a component or a system

Model-level testing can answer a narrow question about behavior under a configured interface. Application testing can establish how retrieval, identity, output handling and tools combine. Neither scope should be reported as the other.

If Kira tests a bare model, the result does not establish that the deployed order service filters records correctly. If she reviews the order API, that does not automatically establish the assistant's handling of misleading text. A complete product review connects the relevant evidence without requiring every exercise to test every layer.

Microsoft's January 2025 lessons paper reports experience assessing both models and applications. Its emphasis on context, downstream impact, automation and human judgment is useful here. Those lessons do not establish a universal ranking of model versus application harms or require a particular tool for every engagement.

Compare methods along useful dimensions

A penetration test, a benchmark and an AI red-team exercise can overlap. A penetration-test scope can include model-mediated behavior. A benchmark can specify an interactive or adaptive evaluator. A red team may use fixed cases as part of its work. Read the actual protocol before inferring coverage from the label.

An adaptive search can reveal a failure that a fixed suite missed. It does not follow that every adaptive rate is directly comparable to every fixed-suite rate. The objectives, budgets, units and scoring may differ. A low benchmark score on prohibited content is not a numerical estimate of internal-note disclosure in this reply workflow.

Use exploration and measurement deliberately

Exploration asks where the requirement can fail and which preconditions matter. It can be manual or automated. A researcher may notice a new shared-draft path; a configured search may find an unanticipated combination of allowed inputs. Both need authorization, effect limits and reliable observations.

Measurement asks a narrower question under a declared protocol. Freeze the configuration and define the unit: one request, a conversation with a budget, or an independently reset task. Separate discovery-selected cases from held-out assessment. Do not add ordinary binomial intervals to adaptive campaign counts without explaining why their assumptions fit.

A single confirmed violation may already establish a defect. Repeating it can characterize reliability and conditions, but repetition is not permission to keep exposing sensitive data or expanding impact. Conversely, a plausible generated order number is not proof of access to a real protected record. Verify the source and reader through controlled evidence.

See attack-success measurement and evaluation statistics for the quantitative work. The first decision is what outcome the numbers will describe.

Work through a partial result

Consider this original synthetic assessment record:

• An internal-note marker appears in a proposed customer reply.
• The reply service denies sending that content to the customer.
• The draft is nevertheless saved in a test folder the customer fixture can read.
• A separate allowed order explanation is accurate and delivered correctly.

The denied send is a successful check on that path. It does not resolve the draft release. If the draft's access record confirms the fixture received the protected marker, report that disclosure and its conditions. If the folder's permissions are known but access evidence is missing, state what is established and what remains unobserved.

Do not count two independent vulnerabilities merely because one source influenced two proposals. Identify the distinct failed controls, consequences and corrective owners. Also preserve the successful order explanation: a fix that blocks every reply may stop the release while failing the product's basic purpose.

Build a finding someone can act on

A finding connects a requirement, observed behavior and material consequence or demonstrated control weakness. For the draft case, record the actor's permitted source control, the source and request versions, the artifact readers, evidence references and the exact boundary that failed.

Separate confirmed root cause from a hypothesis. The observed release may involve draft permissions, missing field filtering or an unexpected consumer. Investigate enough to recommend a control and owner without inventing certainty. A proposed check must cover the actual draft audience, not just the sending tool already shown to deny the operation.

Use the agreed severity method, document uncertainty and identify what would change the rating. “Critical because AI” and “low because it happened once” are both unsupported shortcuts. The operational response can be urgent while the final scope of impact is still being established.

Automation needs a trustworthy assessment path

Choose harness capabilities by the job: reproducible configuration, controlled state reset, bounded attempts, integration with the real checks under test, protected evidence and accountable scoring. A tool's large prompt catalog does not establish coverage of Kira's shared drafts.

Programmatic checks work well for observable states, but their placement matters. A generated proposal, an admitted write and a receiver observation are different signals. Human review is also fallible: use a clear rubric, qualified reviewers and a way to resolve disagreement. A model judge should be validated on relevant examples, including negative and ambiguous cases.

Protect reviewers from unnecessary exposure to disturbing material. Use benign proxy objectives where they answer the question, restricted previews where needed and a clear way to pause or reassign work. This is part of a workable assessment process, not a scoring shortcut.

Close the loop without overstating assurance

Give the product owner the finding, allowed operating scope, evidence gaps, proposed corrective action and retest criteria. Preserve representative safe regression cases where feasible. Some findings also need design review, manual checks or a controlled incident rehearsal; forcing every issue into an automated prompt test can lose the property that mattered.

Report untested components and observation failures explicitly. A clean run establishes its observed result under its protocol, not universal immunity. Configuration, source, audience or workflow changes can invalidate a previous conclusion even when the model name stays unchanged.

Assess the assessment process too: were high-consequence objectives covered, were scorers reliable, and did findings lead to verified changes? Seeded weaknesses in an isolated exercise can reveal a team's blind spots, but finding those seeds is not a complete measure of real-world discovery ability. Independence and diverse expertise support challenge; neither alone guarantees correct judgment.

Produce an assessment brief

For Kira's assistant, write one page containing the protected note rule, actor access, included workflow, excluded systems, allowed test environment and expected evidence. Add one legitimate order question and one denied release case. State which artifact or receipt would turn a suspected draft issue into a confirmed finding, and which result would remain unknown.

Check your understanding