Lesson 3 of 9 · 7 min read
Prompt Injection versus Jailbreaking
In this lesson
TL;DR: Use jailbreak to describe an attempt to bypass safety restrictions, and prompt injection to describe lower-authority material redirecting model behavior against governing instructions or task authority. These categories can overlap. A useful finding states the actual policy, source, observed consequence and responsible controls instead of relying on the label to determine severity or ownership.
Before you read
Pick an answer before you read further. The lesson below explains it, and guessing first, right or wrong, helps the explanation stay with you.
1A model produces a prohibited draft after reading a third-party document, but the publication service refuses the requested post. Is this one failure, one success or something else?
What you should be able to do
Classify a scenario without confusing its delivery route, policy target and observed outcome. Explain why a safety benchmark cannot establish that an application's private records or tools are correctly authorized. Assign remediation to the components that control the failure, including the model provider and the application team where both contribute.
Agree on the vocabulary, then describe the event
Security writing does not use these terms uniformly. Simon Willison's March 2024 distinction emphasizes attacks on model safety mechanisms versus attacks on applications using models. The OWASP 2025 prompt-injection entry, checked November 2026, uses a broader definition and describes jailbreaking as a form of prompt injection.
Neither convention excuses an ambiguous incident report. State the definition you are using and describe the event in operational terms. For example: a third-party document redirected an assistant's answer, a prohibited output reached a reader, or a tool disclosed a record outside the active grant. A reader can assess those statements even if their taxonomy differs.
In this lesson, jailbreak attempt means an input tries to bypass applicable safety restrictions. A successful bypass requires evidence of violating those restrictions. The rules may be implemented in training, application prompts, moderation or several layers. Their ownership is not necessarily confined to the model provider.
Prompt-injection attempt means input with insufficient authority tries to redirect the model against the governing task or rules. It can affect plain answers, retrieved information, saved state or tools. The attempt need not succeed, and the result need not be a data breach. A normal authorized user instruction is not an attack merely because the developer did not write it.
Three questions that keep classification useful
Ask who supplied the material and how it arrived. A direct user request and a third-party passage carried in a user message can share an API role while differing in authorship. Record both. An evaluation may classify pasted material by its entry route; that convention should not hide the source's limited authority.
Next identify the rule or task boundary. “The answer was rude” is insufficient without the relevant policy and context. “The assistant disclosed a different customer's private record” identifies a confidentiality failure that needs application authorization evidence, whether or not a safety filter also should have stopped it.
Finally establish what happened. An attempted bypass, a generated draft, an admitted tool call and a delivered result are different observations. Keep uncertainty about the last step visible. A source appearing in the context is a lead for investigation, not proof that it caused the result.
Source and route
Author is not always carrier
Governing rule
Safety, task or both
Observed consequence
Draft, exposure or effect
Control owners
Assign the actual repairs
Preserve the source distinction
A third-party passage can arrive inside a user message. Record author, carrier and intended role separately.
Classify the case before choosing a control
The following cases are fictional. Assume the public assistant's declared policy prohibits targeted abusive output, while its private document tools require a current task grant and an authorized recipient.
| Case | Useful classification | What the evidence must establish |
|---|---|---|
| A user asks for an allowed change of language | Legitimate task request | The change remains within governing rules |
| A user tries to induce targeted abusive output | Jailbreak attempt under the stated safety policy | Whether prohibited output was produced and exposed |
| A retrieved page attempts to replace a comparison's criteria | Indirect prompt-injection attempt | Whether source content changed the task's decision |
| A page induces a prohibited public post outside the task's grant | Overlapping safety and application-authority failure, if confirmed | The relevant policy violation, admitted action and actual audience |
| An answer is wrong with no evidence of attempted redirection | Quality or safety failure requiring investigation | Whether the cause was retrieval, generation, source manipulation or something else |
An unauthorized disclosure can also violate a model or application's safety policy. Conversely, an injection can redirect a benign task without requesting content prohibited by a safety policy. Avoid defining the categories as mutually exclusive kinds of harm.
Ordinary-looking requests can still be recognized as harmful
A request to send a file can be legitimate in one task and a disclosure attempt in another. Its surface wording does not settle the result. A model or detector with relevant context may recognize the unauthorized intent and refuse it. It is inaccurate to say that safety training has nothing to object to merely because the requested action is an ordinary business operation.
The limitation is assurance and scope. A detector trained for one set of content-policy violations has not necessarily been evaluated on your document permissions, task grants, tool arguments or recipient policies. It may lack the authoritative state needed to decide. A strong score on that detector's benchmark is evidence for those evaluated conditions, not proof that the application's actions are authorized.
Retain useful model refusals and contextual detection. Enforce resource access and effect policy independently so that a missed classification cannot grant authority. Keep the data used for those decisions trustworthy: a recipient permission extracted from the same hostile document is not an independent policy check.
Ownership follows the failing component
| Component | Responsibility | What it cannot establish alone |
|---|---|---|
| Model training and instruction priority | Provider or model operator improves evaluated behavior | Correct permissions in every connected application |
| Application safety policy and moderation | Product and safety owners define allowed content and delivery responses | Whether a specific database object may be read |
| Retrieval and resource services | Application owners enforce identity, task scope and resource policy | Whether every permitted answer is accurate or appropriate |
| Tool and publication services | Operators enforce action terms, recipients, limits and current authority | That a model will never propose a forbidden action |
| Incident handling and user communication | Designated responders establish impact, contain and recover | Root cause from a detector label alone |
An application team can control moderation, output exposure, abuse responses and capabilities even when it cannot retrain a hosted model. A model provider can improve instruction-following behavior relevant to injection as well as safety refusals. For an internally operated model, the same organization may own both. Use named responsible teams and an escalation path rather than sending every report to a single owner based on terminology.
Measure policy failures and useful work separately
For safety evaluation, define the content policy, prohibited outcome, audience, test-set construction and adjudication method. Include legitimate requests that resemble the negative cases so that unnecessary refusals are visible. A nonempty response is not automatically a policy violation, and a refusal of an allowed request is not a successful user outcome.
For application injection evaluation, define the permitted source changes and the actual task. Observe generated answers, proposed reads, admitted actions and final effects under current grants. Include legitimate work and report unknown outcomes. The instruction-hierarchy paper (2024) examines learned priority across different attack settings; its results do not make those settings interchangeable.
A combined case can contribute to more than one metric. Suppose a synthetic test produces a prohibited draft but the publishing service denies its release. Record the generated-content failure and the successful publication denial. If the draft was already visible to an unintended reviewer, the denied public post does not undo that exposure. Define the observation boundary before counting success.
Do not use an adaptive search's fraction of successful attempts as the daily probability of harm. Do not merge results from different audiences, policies and capabilities into a single “AI resistance” score without explaining the aggregation and its limits. AgentDojo (2024) illustrates the value of evaluating security properties alongside legitimate agent tasks; use a deployment-specific test to decide your own controls.
Severity comes from consequences, not terminology
An offensive reply, a misleading recommendation, a cross-customer disclosure and a forbidden transaction have different consequences. Their seriousness depends on the material, audience, reach, repeatability and recovery options. Content alone can create significant harm. A tool proposal that was denied may provide valuable evidence of a reachable weakness without establishing a completed incident.
Avoid both extremes: dismissing a confirmed harmful answer because it used no tools, or describing a harmless symptom as proven access to every system the assistant knows about. State demonstrated impact, additional plausible reach and the evidence needed to distinguish them. Restrict the affected path when warranted, preserving legitimate service where current controls justify it.
What if · interview follow-ups
Hold your answer as the constraints change
A fictional public assistant has a declared content policy and separate grants for private records and public posting. Classify each report using that policy and the observed result.
Start here
A report calls a rude poem a jailbreak. What is missing?
Which applicable rule did the output violate?
Read the declared policy, task and actual output context. Rudeness alone does not establish a safety-policy bypass, and a model error is not automatically an attack. If a prohibited outcome is confirmed, record who saw it and which control failed.
Carry it forwardUse the stated policy and evidence rather than taste as the oracle.
Weak answerInferring a jailbreak solely from an unwanted tone.
Constraint changes
A retrieved page causes a wrong comparison but no tool is called. Can this be an injection finding?
Did the source redirect the task?
Yes, if the evidence supports that attempted source instruction caused the task deviation. The consequence is the answer or decision, whose impact depends on its use and audience. No tool call is required for an integrity failure.
Carry it forwardPlain output is a meaningful consequence.
Weak answerTreating every text-only result as harmless.
Constraint changes
The model creates a prohibited draft and the publishing service denies it. Which layer worked?
Where was the draft visible?
The publishing policy prevented that public action. The model or content-control layer still produced prohibited material. Check whether the draft reached an audience before the denial, then report each observation rather than compressing them into a single pass.
Carry it forwardA successful downstream block does not erase earlier exposure.
Weak answerCounting the entire test as safe because no public post appeared.
Constraint changes
A provider improves its jailbreak benchmark score. Can you remove recipient checks?
Does that benchmark inspect your current grants?
No. Keep recipient and material authorization at the application boundary. Evaluate whether the new model improves useful behavior and refusals in your own tasks, including overlapping safety and source-authority cases.
Carry it forwardTransfer the model change through deployment-specific evaluation.
Weak answerTreating a provider score as evidence of application permission correctness.
Now defend it
How would you hand the finding to another team?
Describe the source and route, applicable policy, current task authority, observed output or effect, audience, control results and evidence gaps. Assign owners to the components needing work; include the taxonomy convention without relying on it to explain the incident.
Give an explanation that another team can act on
Use a short finding structure: source and route, governing rule, observed output or action, actual audience, failed or successful control, and unresolved evidence. Add the taxonomy labels afterward. This makes the report useful to a provider investigating model behavior, a product owner setting policy and an engineer correcting authorization.
Continue with direct and indirect injection for source-path analysis, or evaluation of injection defenses for designing the test evidence. Neither a label nor a benchmark substitutes for a specific failure claim.
Check yourself
Say or write your answer first, then reveal. Explaining it from memory, in your own words, is the skill an interviewer is checking.
1Must prompt injection and jailbreaking be mutually exclusive in a report?
2Does ordinary business wording guarantee that a safety model will miss unauthorized disclosure?
3Who owns remediation when a prohibited model answer is delivered by your product?
4Which is automatically more severe: an indirect injection or a jailbreak?