PyRIT Explained: Microsoft's AI Red Teaming Toolkit After 1.0, and the Foundry AI Red Teaming Agent

PyRIT is the Microsoft AI Red Team's open-source toolkit, and it is aimed at a different job from a scanner. Its README describes it as a toolkit for finding security and safety weaknesses through "automated scans, adaptive multi-turn attacks, and human-led testing". Where a scanner runs a fixed test set, PyRIT lets a red teamer build attacks in which an adversarial model converses with the target and adapts to its replies.

What it is

PyRIT ("Python Risk Identification Tool for generative AI") lives at microsoft/PyRIT under the MIT licence; the older Azure/PyRIT repository is archived. Version 1.0.0 was released on 24 July 2026, and later 1.x releases followed. It needs Python 3.11 or later. Microsoft also builds on PyRIT in the AI Red Teaming Agent in Microsoft Foundry, which turns some of its capabilities into a managed scan with a scorecard.

How it works

The current framework has these building blocks:

• Targets are what you send prompts to: model APIs, HTTP endpoints, WebSocket and browser-automation targets, or custom code.
• Converters transform prompts, for example encoding, translation, or turning text into images or audio. They can be stacked.
• Scorers judge responses as true or false, on a Likert scale, by classification, or with custom logic.
• Attacks run single-turn or multi-turn conversations between an adversarial model and the target. Named multi-turn techniques include Crescendo and TAP.
• Scenarios package datasets and techniques so they can run at scale.
• Memory stores every prompt, response and score, in memory, in SQLite or in Azure SQL.

The 1.0 break. The 1.0 release notes list several breaking changes:

• scenario "strategies" became "techniques"
• components moved to registries
• PromptConverter became Converter
• conversation state moved to a new model
• pre-1.0 compatibility paths were removed
• the memory schema changed

Code written for the older orchestrator-based API needs updating.

How to use it

Day one: configure and list scenarios

Install the pyrit package from PyPI and put provider keys in ~/.pyrit/.env, which PyRIT reads at start-up. Then list the packaged scenarios:

pyrit_scan --config-file ./.pyrit_conf list-scenarios

Week one: run a packaged scan against your own target

PyRIT's documentation shows running a scenario against a configured target with a chosen technique, for example:

pyrit_scan run foundry.red_team_agent --target openai_chat \
    --initializers target --techniques base64

Use pyrit_scan for repeatable batch runs and pyrit_shell for interactive exploration and comparing runs. Scenario families cover areas such as jailbreak resistance, data leakage, multilingual robustness, and a set adapted from garak's probes.

Every day: validate the scorer, then trust the numbers

Before reporting attack success rates, check the scorer on a sample of responses that people have labelled. A scorer that calls refusals "successes", or misses partial compliance, makes every downstream number wrong. Keep the memory database as evidence, and turn confirmed findings into regression tests.

The managed option: Foundry's AI Red Teaming Agent

In Microsoft Foundry, the AI Red Teaming Agent runs automated scans for content risks and, for Foundry-hosted agents, agent-specific risks such as prohibited actions, sensitive data leakage, task adherence and indirect prompt injection. It reports attack success rate, the share of attacks that succeeded. Microsoft documents its limits: it supports only text scenarios, does not support agents hosted outside Foundry, uses synthetic data and mock tools, and its success rates come from generative models, so they can vary between runs and include false positives.

Where it stops

• A framework, not a verdict. PyRIT gives you building blocks; the quality of a red team depends on the objectives, the techniques chosen and the scorer.
• Scorers are models. Automated scoring needs validation against human labels.
• API churn. The 1.0 changes mean older blog posts and internal scripts may not run.
• Authorisation still applies. Test only systems you own or are authorised to test, and handle any harmful content generated during testing as sensitive data.

How it comes up in interviews

• "Design an automated red-teaming loop for a customer-facing assistant." Objectives from the threat model, multi-turn attacks, validated scorers, stored evidence, regression tests.
• "Your attack success rate went from 3% to 9% after a model update. What do you check?" Scorer drift, sampling variance, dataset changes, then real regressions.
• "Why are multi-turn attacks harder to defend against?" Each turn can look harmless; the risk is in the trajectory.

Sources

Checked on 13 November 2026: PyRIT GitHub README (microsoft/PyRIT) and v1.0.0 release notes (24 July 2026); PyRIT framework and scanner documentation; PyPI pyrit project page; Microsoft Learn page on the AI Red Teaming Agent in Microsoft Foundry.