Lesson 1 of 9 · 5 min read
Agentic AI Security
In this lesson
TL;DR: An agent proposes actions; your application decides which actions can execute. Assume that untrusted content or a model error can influence a proposal. Enforce task permissions, approvals and resource limits in code outside the model, and verify that no alternate tool or credential bypasses them.
After this lesson, you should be able to draw the boundary between a model proposal and an authorized action, choose controls for a small support agent, and explain what a stop mechanism must actually stop. If needed, first read tool calling and the agent loop.
Before you read
Pick an answer before you read further. The lesson below explains it, and guessing first, right or wrong, helps the explanation stay with you.
1A document-summary agent can delete any file the employee owns. Its prompt says never to delete anything. What would you change first?
From an answer to an action
A text-only assistant can give a wrong answer or disclose information in its response. Tool access adds side effects: sending a message, changing an order, fetching a URL or executing code. Those operations use application credentials and network access, not powers inherent in the language model.
Suppose an outsider can write a support ticket. The legitimate task is to summarize the ticket and propose a resolution. If the ticket influences the model to request an unrelated refund, the ticket author has crossed from supplying content into influencing an action. The security design must keep that influence from becoming authority.
Inspect four parts of the system:
- Inputs: user requests, retrieved documents, messages, tool descriptions and results. Identify who may write each input and what authority, if any, it carries.
- Operations: what each tool can read, change, send or execute. A tool's name is less informative than its implementation and credentials.
- State: memory, generated files, queued work and results passed between agents. Influence can persist beyond one conversation.
- Execution: the service that checks permission and commits a side effect. This is where a proposed action becomes real.
External content
Can influence reasoning
Agent loop
Proposes a tool call
Policy gateway
Checks identity and task
Resource service
Enforces object access
State the enforcing services must check
Task scope
- Authenticated user and agent
- Allowed operation and resource
Action approval
- Exact recipient, amount or file
- Expiry and changes checked again
Current resource state
- Object ownership and permission
- Caps, concurrency and retries
Influence is not permission
A document may influence what the model proposes. Its author does not gain authority over the resource service.
Worked review: a customer-support assistant
Assume the assistant can read a selected ticket and order, draft a reply and request a refund. A customer controls the ticket text. The assets are customer records, payment integrity and the ability to stop unintended activity.
Before reading the table, choose the first check you would add if the refund tool currently accepts any order identifier. Then identify a distinct risk that the same check would leave open.
| Decision | Enforcement point | What remains |
|---|---|---|
| Read only records authorized for this employee and task | Ticket and order services use authenticated identity and selected task scope | Authorized text may still mislead the model |
| Refund only an eligible order, with cumulative limits | Refund service checks ownership, prior refunds and policy in one atomic operation | An incorrect refund within the allowed limits is still possible |
| Keep email draft-only until sending is required | Tool gateway exposes no send action; sandbox has no alternate sending credential | The user may still approve a misleading draft later |
| Protect configuration from the task | Agent identity cannot write policy, tool registration or startup configuration | Approved dependencies and administrators still need review |
| Bound cost and duration across delegated work | Host tracks shared task budgets and stops new dispatch | Already committed actions may require reconciliation |
Worked reasoning. Order authorization comes first: a valid identifier is not proof that the task may refund it. Derive the permitted order from trusted state and reject a different identifier in the refund service. Then add cumulative limits and retry handling. Authorization alone does not prevent repeatedly refunding the same eligible order. Excessive agency develops that second decision.
The lethal trifecta helps inspect disclosure routes through mail, browsers and shared outputs. It does not cover every financial or destructive action; use it alongside the action inventory.
Make each layer's claim precise
Authorization prevents a defined operation. A server checks the authenticated subject, resource and action before returning data or changing state. It must cover every route to that resource, including retries and background workers. A correct check does not compensate for a separate endpoint that skips it.
Isolation limits reachable resources. Run generated code and local tools with the filesystem, network and credentials they need for the task. Verify those limits from inside the sandbox using harmless fixtures. Isolation has configuration and implementation assumptions; do not present the word sandbox as a proof.
Approval authorizes a particular action. Show the real recipients, content, amounts and objects from trusted code. Bind approval to them and check again at execution. Approval cannot grant permissions the user lacks, and it can still be mistaken. Read human-in-the-loop design.
Detection supports response. Keep caller and agent identity, operation, policy result, time and relevant source references. Restrict access and retention. Preserve sensitive content only when justified and protected; logging every prompt can create another data exposure. Do not treat a model-generated explanation as a reliable record of why it acted.
A stop mechanism contains further activity. Test that it disables new calls, queued work, delegated agents and usable credentials. It cannot undo an email already delivered or a payment already committed. Design reconciliation and recovery for those cases.
A classifier may run in a separate service and still be probabilistic. Conversely, an authorization rule is deterministic only with respect to its inputs and implementation. Judge the mechanism, not whether a box is drawn inside or outside the model.
Use frameworks to check coverage
OWASP released its Top 10 for Agentic Applications in December 2025. Four useful mappings for this example, checked November 2026:
| Design concern | OWASP category | Question to resolve |
|---|---|---|
| A ticket changes the requested task | ASI01, Agent Goal Hijack | What prevents that change from authorizing an action? |
| A tool carries broad credentials | ASI03, Identity and Privilege Abuse | Which user and task bound this call? |
| Ticket-derived content persists | ASI06, Memory and Context Poisoning | Is its source preserved across sessions? |
| Review hides the real consequences | ASI09, Human-Agent Trust Exploitation | Can the person inspect the actual action? |
These are a coverage aid, not a complete review or a ranking of this product's risks. Use the full framework for additional concerns such as tool supply chains, communication between agents and cascading failures. Write the concrete failure and enforcement point before assigning its category.
Change one assumption
Attempt each decision before revealing the explanation.
What if · interview follow-ups
Hold your answer as the constraints change
A support assistant reads tickets and order details, proposes refunds, and drafts customer replies. You are reviewing its permissions before enabling automatic actions.
Start here
What if the employee can access every customer, but this task concerns one order?
Is the employee permission set the right task scope?
Use the employee permissions as an upper bound, then narrow the task to the selected order and allowed operations. Derive that selection from trusted application state. The refund service must enforce it even if the model supplies a different valid order identifier.
Carry it forwardTask authority should be narrower than the user authority it comes from.
Weak answerGiving each task everything the employee can access.
Constraint changes
What if a tool asks the assistant to install an extra server to finish the task?
Who is authorized to change the tool set?
Treat that request as untrusted tool output. A separate configuration workflow must review the new server, credentials and execution permissions. The agent must not gain broader authority merely by following a dependency suggestion. A tool result cannot approve its own installation.
Carry it forwardKeep authority changes outside the component being constrained.
Weak answerLetting a task add tools and credentials without review.
Constraint changes
What if users approve messages without reading them?
Can the product remove the send operation for some tasks?
Keep summaries and drafts automatic within their read scope. Use review for a concrete send with actual recipients and attachments, and apply data-sharing policy regardless of approval. Measure whether people understand the decision. A classifier can help triage, but cannot replace the underlying permissions.
Carry it forwardReduce unnecessary decisions while retaining checks at consequential actions.
Weak answerTreating a high approval rate as proof that the actions were safe.
Constraint changes
What if the assistant delegates work to another agent?
Does a new agent inherit a new permission boundary?
Pass a bounded task and identify both the requesting user and acting agent. The receiving service must not replace the task scope with its own broader service account. Treat returned text and memory as data with provenance, and apply shared budgets so delegation cannot reset the limits.
Carry it forwardDelegation must preserve or narrow authority and account for shared limits.
Weak answerAssuming each agent is safe because it runs in a different process.
Now defend it
What evidence would you require before enabling automatic refunds?
In an isolated test, verify cross-customer rejection, atomic cumulative caps, duplicate handling and approval binding. Exercise the stop mechanism against queued work and credentials. Review minimized audit records with the incident responder and identify the owner of any remaining financial risk.
Check whether you can explain the design
Describe the legitimate task, one attacker-controlled input and one unauthorized operation. Point to the component that rejects it even when the model asks for it. Then name a remaining risk and a test that could reveal a bypass. Use that explanation as your starting point for agent security questions.
Check yourself
Say or write your answer first, then reveal. Explaining it from memory, in your own words, is the skill an interviewer is checking.
1The model has been persuaded to request a refund for a different customer. Which component must reject it?
2Why can a classifier running outside the main model still be probabilistic?
3The stop button ends the chat. What else must an incident-response drill check?