Trust Boundaries, Assets and Adversaries

TL;DR: Before naming any control, name the asset (what loss would hurt), the adversary (who wants it and what they can actually do), and the trust boundaries (every place where data or commands move between parties that trust each other differently). A boundary review asks whether less-trusted input can exercise authority it should not have, and which component prevents that. AI systems add boundaries that are easy to miss, because a language model reads text from many authors in one context.

Before you read

Pick an answer before you read further. The lesson below explains it, and guessing first, right or wrong, helps the explanation stay with you.

1. 1A company encrypts every database with AES-256 and calls its customer data 'secure'. An engineer with read access exports the table to a personal laptop. Did the encryption fail?

   Commit to a guess, then reveal

What you should be able to do

By the end of this lesson, draw a small system and annotate each crossing with three things: who controls the input, which authority the receiving component has, and the check that limits what the input can cause. Explain why encrypting a database does not stop an authorised reader from exporting it. Then apply the same reasoning to an agent that reads customer tickets and calls a CRM.

You do not need cryptography or model-training knowledge to start. You need to be able to follow a request from a browser to a server and then to a database. The first diagram makes that path explicit.

Why security reasoning starts with nouns

“Add encryption, MFA and input validation” leaves a design review unfinished. Encryption may protect a stolen disk; MFA may protect a sign-in; a parser may reject a malformed request. None tells you whether one customer can retrieve another customer’s records. Start by naming the loss you want to prevent and the capabilities that could cause it. Then select a control at the point where those capabilities meet the asset.

Three nouns carry most of the weight.

The asset. What would hurt if it were read, changed, destroyed or made unavailable? Customer records and payment data are obvious. Less obvious assets are often worth more: signing keys, model weights, the ability to send email as the company, an administrator's session, the integrity of a training dataset. A useful test is to ask what an attacker would sell, ransom or use as a stepping stone.

The adversary. Who wants the asset, and what can they actually do? "Hackers" is not an adversary model. A useful description names a capability: an anonymous internet user who can send requests to your API; a customer of another tenant on the same platform; a contractor with read access to one repository; someone who can publish a web page your agent will read. Capability decides which controls matter. Encryption at rest is decisive against disk theft and irrelevant against a logged-in insider.

The trust boundary. A trust boundary is any place where data or control passes between components that trust each other differently: browser to server, service to database, your code to a third-party library, a user's document to your language model. Every boundary is a question: does anything check what crosses here, and is the check enforced by the side that needs protecting?

Describing the adversary by capability

"Attackers" becomes useful once it is broken into capabilities. Four dimensions cover most designs:

The fourth row is the one older threat models leave out and AI systems make central. Someone with no account and no network position at all can still be an adversary if they can write text your agent will later read.

Find where input gains authority

Look at the classic vulnerability classes and the same shape repeats. Data crosses from a less trusted side to a more trusted side and is treated as if it came from the trusted side.

• SQL injection. User input crosses into the database's query language and is interpreted as code.
• Cross-site scripting. One user's input crosses into another user's browser and runs as the site's script.
• Server-side request forgery. A user-supplied URL crosses into the server's network position, which can reach internal services the user cannot.
• Deserialisation flaws. Bytes from outside cross into an object loader that can execute code.
• Prompt injection. Text written by a third party crosses into a language model's context and is followed as an instruction.

These examples share a review question: where can input gain authority? For prompt injection, instructions and external text enter the model’s context together. A prompt delimiter does not enforce the separation that a parameterised database query provides. Treat the model’s proposed actions as requests to an independently enforced policy; the later prompt-injection lesson develops that distinction. Boundary analysis is a useful lens, not a claim that every vulnerability has the same cause.

The formal name for the bug pattern is CWE-501, Trust Boundary Violation: the product mixes trusted and untrusted data in the same data structure or message. A session object that holds both the authenticated user ID and a value copied from a request header, or a prompt that holds both instructions and a retrieved document, are both instances. Once mixed, later code cannot tell which parts were checked.

Drawing the boundaries in practice

You do not need a formal method to start. Sketch the components a request touches, draw a box around everything one party controls, and mark each arrow that crosses a box. Then, for each crossing, write one line: who could control what crosses, and what checks it. Gaps show up immediately.

Three habits make the sketch honest.

Ask who can write to each input. This single question finds more AI security bugs than any other. A support bot that summarises tickets reads text written by every customer. A coding agent that reads issues reads text written by anyone with a GitHub account. A retrieval system reads every document anyone could upload. Each of those authors is on the far side of a boundary, even when the data sits in your own database.

Put checks on the side that needs protection. A check in the browser protects nothing, because the attacker controls the browser. A rule in a system prompt protects nothing reliable, because the attacker can write text into the same context. Enforcement belongs in code the attacker cannot influence: the server that owns the data, the API that moves the money.

Notice boundaries that look internal. Microservices calling each other, a nightly job reading a shared bucket, an agent passing results to another agent: each is a crossing if the parties differ in what they are trusted to do. Attackers move by finding the internal crossing nobody checks.

Rules that keep a diagram honest

Microsoft's 2006 article that popularised STRIDE (Hernan, Lambert, Ostwald and Shostack) gave four sanity rules for data-flow diagrams that still catch most drawing mistakes:

1. No magic sources or sinks. Data comes from somewhere and goes somewhere; a store nothing writes to or a flow from nowhere hides an author.
2. No psychokinesis. Data never moves from a person to a store without a process in between, because that process is where checks live.
3. Collapse similar elements inside one boundary, so the diagram stays small enough to review.
4. Use a context diagram plus breakouts rather than one enormous picture.

The same article made two points that experienced reviewers repeat. Boundaries depend on deployment, not just design: a flow from an application to its database is internal when they share a host and crosses a boundary the day the database moves to another machine or a managed service. And threats often appear only when systems are composed: two components that are each safe can be combined so that one's assumptions about its input are violated by the other's output.

Network location is not a trust boundary: zero trust

The idea that "inside the network" means trusted is the assumption attackers rely on after the first foothold. NIST SP 800-207, Zero Trust Architecture (August 2020), turns its rejection into seven tenets. For this design review, apply these principles:

• All communication is secured regardless of network location.
• Access to each resource is granted per session; authenticating to one resource does not grant access to another.
• Access is decided by dynamic policy that considers the identity, the device and other signals.
• No asset is inherently trusted; the enterprise monitors the posture of everything it owns.

SP 800-207 describes the enforcement machinery as a policy engine that makes each access decision, a policy administrator that sets up or tears down the session path, and a policy enforcement point in front of the resource that allows, monitors and ends connections. CISA's Zero Trust Maturity Model 2.0 (April 2023) organises the same idea into five pillars: identity, devices, networks, applications and workloads, and data.

Where AI systems hide extra boundaries

Traditional systems keep code and data in different places, so boundaries are visible in the architecture. A language model application folds many authors into one artifact, the context window, and that hides boundaries inside a single string.

The table is the starting point for every agent design review you will do in this curriculum. The concepts on why prompt injection exists and the lethal trifecta build directly on it.

Worked example: review a support assistant

Consider an illustrative support system. A customer submits a ticket through a web UI. An orchestrator reads the ticket, retrieves knowledge-base passages, sends selected context to a hosted model and uses a CRM tool to propose an update. Employees and contractors can edit the knowledge base. These are assumptions for the exercise, not observations about a particular company.

Before choosing controls, write down three assets: customer records, the authority to change a CRM record and the confidentiality of the context sent to the model provider. Then review the crossings:

A store being internal does not make every author trusted. The ticket text and retrieved passages remain data, even when the orchestrator reads them from a company database. A valid tool call can also request an unauthorised operation: correct JSON says nothing about whether this caller may update that record.

Make a decision. Put customer-level authorisation in the CRM tool or its backing API. Derive the caller identity from the authenticated session; do not let a model-supplied customer ID establish permission. Require confirmation for consequential changes, while keeping the permission check in place even after confirmation.

Check the decision. In a local test fixture, create two customers and a record belonging to each. Verify that the first customer’s request cannot read or change the second customer’s record through the tool. Inspect what enters the model context as well: blocking the final answer is too late if the other customer’s data has already been sent to a provider. Run these checks only against systems you own or are authorised to test.

Change one condition. Suppose contractors can now edit the knowledge base. The CRM permission rule should still hold. What changes is your input provenance and ingestion review, not the identity used to authorise the caller’s records. A useful boundary map lets you explain both parts.

Check your assumptions

• "Internal network means trusted." SP 800-207's second tenet says the opposite.
• "Zero trust is a product you buy." It is an architecture: per-request decisions from identity and posture.
• "Data in our own database is trusted." Trust follows the original author, not the storage location.
• "We validate in the client." The attacker controls the client; checks belong on the protected side.
• "Boundaries are fixed once designed." Redeployment and composition move them.

What if · interview follow-ups

Hold your answer as the constraints change

You are reviewing a company's internal platform: services in one cloud VPC, a shared database, a nightly batch job that reads a partner-writable bucket, and a new agent that reads web pages and updates the CRM. The interviewer asks where the trust boundaries are, one component at a time.

1. 1

   Start here

   What if all services run inside one VPC; is there a boundary between them?

   Nudge meAnswer it out loud, then reveal

Revising, not practising? Show every answer

How this shows up in interviews

In a design exercise, make your reasoning inspectable. Start with “Let me list who can write to each input and what the system can reach.” Draw one crossing, name the permission that could be misused and place a check on the receiving side. If a requirement changes, update the map before adding another control. That gives the interviewer a specific decision to examine; it does not assume a universal scoring rubric. The threat modeling lesson turns this habit into a repeatable method.

Say it in 30 seconds: "I start every review with three things: the asset, the adversary and their real capability, and every trust boundary where data or control changes hands. I look for places where less-trusted input can gain authority and check what enforces the limit. In LLM systems the context window mixes authors, so I ask who can write to each input, and I put enforcement in code on the side that needs protecting, never in the prompt."

Check yourself

Say or write your answer first, then reveal. Explaining it from memory, in your own words, is the skill an interviewer is checking.

1. 1An internal HR assistant answers employees' questions using a vector index of company policy documents. Only HR staff can upload documents. Is retrieved text trusted?

   Answer it, then reveal

2. 2Where should the rule 'only managers can see salary data' be enforced in an LLM application?

   Answer it, then reveal

3. 3Name three of NIST SP 800-207's zero-trust tenets.

   Answer it, then reveal

4. 4Why can moving a database to a managed service change the threat model even if no code changes?

   Answer it, then reveal

Sources and further reading

• OWASP Threat Modeling Cheat Sheet, especially system modeling and review: supports the use of data flows, actors, stores and trust boundaries, followed by testable mitigations.
• NIST SP 800-207, Zero Trust Architecture, sections 2.1 and 3: the source for the zero-trust tenets and the policy engine, administrator and enforcement-point roles. The support-assistant exercise is an application of those ideas, not an example quoted from NIST.