AI security glossary

An input deliberately crafted to make a model produce a wrong or attacker-chosen output, often through changes a human would not notice. Adversarial suffixes found by optimisation, such as GCG, are the language-model version.

A record of the components of an AI system: base models and their revisions, datasets, training code, dependencies and licences. It extends the software bill of materials to models and data, and supports model provenance.

Structured adversarial testing of an AI system to find security and safety failures before attackers or users do. Results are reported as success rates over repeated attempts because model behaviour is stochastic. See how it differs from penetration testing.

Hidden behaviour implanted in a model, usually through poisoned training or fine-tuning data, that activates only when a specific trigger appears in the input. The model behaves normally otherwise, so standard evaluations miss it.

A program with legitimate authority that is tricked into using it on an attacker's behalf. An agent holding credentials broader than the requesting user's is a confused deputy waiting to happen.

Manipulating training, fine-tuning or retrieval data to change how a model behaves, either broadly or through a targeted backdoor. OWASP lists it as LLM04, Data and Model Poisoning.

An attack that drives up inference cost or exhausts capacity through expensive requests, long outputs or agent loops. Part of OWASP LLM10, Unbounded Consumption. See the worked question.

A mathematical guarantee that a model's outputs change only slightly whether or not any single record was in its training data. Differentially private training limits membership inference and memorisation, at some cost in accuracy.

Prompt injection in which the person interacting with the application supplies the malicious instructions themselves.

A design proposed by Simon Willison in 2023 in which a privileged model plans and calls tools but never sees untrusted content, while a quarantined model processes that content and returns results as opaque references.

An LLM system holding more functionality, permissions or autonomy than its task requires, so that a manipulated model can cause outsized harm. OWASP LLM06. See the worked question.

A control placed around a model to check or constrain its inputs and outputs, such as a classifier, a policy filter or a schema validator. Probabilistic guardrails reduce risk; they do not create a security boundary.

Passing model output to a browser, database, shell or other interpreter without validation or encoding, recreating injection vulnerabilities such as cross-site scripting. OWASP LLM05.

Prompt injection delivered through content the model processes on the user's behalf, such as a web page, email, document or tool result, rather than typed by the user. The main entry point for attacks on agents.

A training approach that teaches a model to give system instructions priority over user instructions, and both priority over text found in tool outputs. It lowers injection success rates but does not guarantee separation.

An input that makes a model produce content its safety training is meant to prevent. Unlike prompt injection, it targets the model's own policies rather than the application built on it.

Simon Willison's term for an agent that combines access to private data, exposure to untrusted content and the ability to communicate externally. Together they make data exfiltration through prompt injection possible. See the worked question.

A long-context attack that fills the prompt with many fabricated examples of an assistant complying with harmful requests, so the model continues the pattern. Described by Anthropic in 2024.

An open protocol, introduced by Anthropic in 2024, for connecting models to tools and data through servers that advertise tools, resources and prompts. Tool descriptions from MCP servers enter the model's context, which creates the risk of tool poisoning.

An attack that determines whether a specific record was part of a model's training data, usually by exploiting the model's higher confidence on training examples. See the worked question.

Getting an agent to store an injected instruction in persistent memory so that it influences later sessions, turning a single injection into a lasting compromise.

A knowledge base of adversary tactics and techniques against AI systems, with case studies, structured like MITRE ATT&CK. Used for threat modeling, red team planning and detection mapping.

Using query access to copy a model's behaviour, through distillation on its outputs, or to recover some of its parameters. See the worked question.

The OWASP GenAI Security Project's list of the most important risks in LLM applications. The 2025 edition runs from LLM01 Prompt Injection to LLM10 Unbounded Consumption.

An attack in which crafted input causes a language model to follow the attacker's instructions instead of the developer's, possible because instructions and data share one context. OWASP LLM01. See direct versus indirect.

A tool or MCP server that changes its definition or behaviour after the user approved it, so the approval no longer describes what runs. Mitigated by pinning versions and requiring re-approval on change.

A model weight format that stores tensors and metadata without executable content, so loading it cannot run code, unlike pickle-based formats. See unsafe model files.

A family of techniques, described by Microsoft researchers in 2024, that mark untrusted input with delimiters, interleaved tokens or encoding so the model can better tell it apart from instructions. It reduces injection success but is not a boundary.

Disclosure of the instructions given to a model. Harmful mainly when the prompt contains secrets or is used to enforce access control, neither of which it should. OWASP LLM07.

Hiding instructions in a tool's name, description or schema, which the model reads as context, so that it misuses that tool or others. A main risk of third-party MCP servers. See the worked question.

Recovering verbatim training data, including personal information or secrets, from a model's outputs. Memorisation increases with model size and with duplication of the data.