garak Explained: NVIDIA's LLM Vulnerability Scanner, Probes and Detectors, and How to Read a Report

garak is the closest thing the LLM world has to a network vulnerability scanner: point it at a model or an LLM-backed endpoint, choose families of tests, and get a report of where it failed. Its own description is that it "checks if an LLM-backed application can be made to fail in a way we don't want." It is a common answer to "how would you automate LLM red teaming?", and a good way to show you understand what automated scanning can and cannot prove.

What it is

garak is an open-source Python tool maintained under NVIDIA's GitHub organisation and licensed under Apache-2.0. As of November 2026 the latest release is v0.17.1, which requires Python 3.11 or later. It works against many kinds of target: hosted model APIs, Hugging Face models, local models and REST endpoints that wrap an application.

How it works

garak is built from plug-in components:

• Generators connect to the target.
• Probes generate interactions designed to provoke a particular failure, such as leaking a system prompt, following instructions hidden in encoded text, or producing toxic content.
• Detectors decide whether each response shows the failure.
• Harnesses structure the run; the default is probewise.
• Evaluators produce the scores.

Each probe and detector pair gives a failure rate over the attempts.

How to use it

Day one: install and list what is available

python -m pip install -U garak
garak --list_probes            # also --list_detectors, --list_generators

Provide the target's credentials as environment variables, for example an API key for a hosted model, and only point garak at systems you own or are authorised to test.

Week one: run a focused scan

python3 -m garak --target_type openai --target_name <your-model> \
    --spec probes.encoding --generations 5 --report_prefix encoding-baseline

--spec selects probes (it replaced the deprecated --probes, --probe_tags and --buffs options). --generations sets how many responses to sample per prompt; more samples catch failures that only happen sometimes. For remote models, raise --parallel_attempts to finish in reasonable time. Start with the probe families that match your threat model rather than running everything.

Reading the results

• Console lines show each probe and detector pair with a count, such as "ok on 17/20", and a failure rate.
• The report file, garak.<run id>.report.jsonl, records every attempt and is updated while the run proceeds.
• The hit log contains only the attempts that produced a failure. Read these first, because they show the actual prompt and response.
• The HTML summary compares your target with a calibration set of recent models using Z-scores. A positive score means better than average. A high Z-score with a low absolute pass rate still means the target did poorly; the other models just did worse.

Every day: make it a regression test

Save a baseline per model version. Re-run the same --spec selection on every model, system prompt or guardrail change, and alert when a failure rate rises. Feed confirmed failures into your own test suite, so they become permanent checks rather than one-off findings.

Where it stops

• Coverage is the probe set. A clean run proves only that these probes, with these detectors, did not find a failure.
• Detectors make mistakes. String matchers and classifiers can miss failures or flag harmless output; read the hit log before reporting.
• Model, not system. Scanning a bare model says little about an application's tools, retrieval and permissions, where the serious agent risks live.
• Sampling matters. Few generations per prompt under-report failures that happen only some of the time.

How it comes up in interviews

• "How would you add automated LLM red teaming to a release process?" Pick probes from the threat model, baseline them, re-run on every change, gate on regressions, and keep manual testing for what automation misses.
• "garak shows 0% failure on prompt injection probes. Ship it?" Explain what that does and does not cover, and what you would test manually.
• "garak, PyRIT or promptfoo?" garak for broad scanning against a target; PyRIT for building adaptive multi-turn attacks; promptfoo for evaluation-driven testing in CI.

Sources

Checked on 13 November 2026: garak GitHub README (NVIDIA/garak) and releases; garak CLI reference and reporting calibration documentation (reference.garak.ai); garak documentation on reading results (docs.garak.ai); PyPI garak project page.