Prioritize attack paths using reach, exposure and evidence

TL;DR: Prioritize a path by the harm it can cause, the conditions needed to reach that harm and the evidence that those conditions exist. Give every decision an owner and a test. A risk ranking is specific to the design; a universal list of “worst AI attacks” cannot decide what this feature should ship.

Where you are. Module one, lesson two. You will turn the boundary map into a risk register and choose the next engineering task.

Work from a small evidence packet

Use the fictional assistant from lesson one. A design review has established four facts:

• Customer chat can query tickets using a tenant filter, but it has no customer-level filter.
• Staff drafts contain clickable links. The product team has not decided whether images or automatic previews will be supported.
• Refund execution is not connected yet. Only a proposed amount appears in the staff interface.
• Prompt and response traces are available to a broad internal operations role; their retention policy is unresolved.

These are exercise facts. They make one priority clear: customer chat can return another Alder customer's ticket. The flaw does not require prompt injection. A tenant filter protects Alder from Birch while leaving Mina and Noor exposed to one another.

Now distinguish known exposure from plausible future exposure. Refund execution would add financial authority, but it is not an active path in this version. Rich-output disclosure needs a renderer and destination path; those are design questions to resolve before enabling the feature. Unrestricted traces are already another route to private data.

Write a path that can be challenged

Use the form: actor, input or capability, failed boundary, affected asset, required conditions. “Prompt injection” alone supplies none of those details. A better entry is: an Alder customer queries shared ticket retrieval; missing customer authorization permits Noor's ticket to enter Mina's answer; the affected asset is another customer's private support data.

Keep the condition in the sentence. If a mandatory object check removes unauthorized records before any prompt, log or response receives them, that particular path is interrupted. If it runs only in one UI route, a worker or export path may still bypass it.

The table is a decision record, not a measured incident-frequency model. Avoid numerical likelihood scores that suggest more precision than the evidence supports. An attacker can write a ticket; whether it is retrieved, acted upon and delivered across a boundary still depends on the system.

Name a control and its refusal test

For the first finding, the invariant is: Mina's customer session cannot expose Noor's private ticket through retrieval, context construction, cache reuse or answer delivery. The retrieval owner must show where current customer permissions are checked. The test uses synthetic records with distinct owners and a query that strongly matches Noor's record.

For the trace finding, “encrypt the logs” is incomplete. Encryption at rest does not stop a logged-in operations user from reading the data. Remove unnecessary bodies, restrict readers and retention, and verify that errors do not copy private text into a less protected log. The owner and evidence differ from the retrieval fix.

A control can prevent one effect while leaving others open. Disabling remote images removes that browser request channel. It does not establish that a plain-text draft contains only data its recipient may see. A schema can restrict a proposed refund's shape while leaving the wrong order eligible. State the effect the control actually enforces.

Let a changed constraint change the ranking

Suppose a scheduled worker is added that executes approved refund proposals. The refund path now deserves immediate review: identity, cumulative balance, approval binding, retries and uncertain outcomes become release conditions. The previous low priority was based on absent execution authority, not on refunds being intrinsically low risk.

Now suppose customer chat uses published help articles only, with no private ticket access. The ticket-disclosure path may no longer be reachable from that feature, but staff draft and trace paths remain. A reduced feature can be a legitimate release choice if the removed access is enforced and the limitation is clear to users.

System instructions can improve behavior and belong in evaluation. They do not replace the object or delivery checks above. Prompt confidentiality also has a context-specific value: do not store secrets there, but assess any business-sensitive instructions rather than declaring every prompt leak harmless or inevitable.

OWASP's threat-modeling guidance supports tracing threats to mitigations and validation. Our register adds explicit evidence gaps so an unverified assumption cannot quietly become a release claim.

Do this before moving on

Produce four entries for this case. Each needs an actor, boundary, asset, evidence, owner, control and refusal test. Pick the first engineering change and explain why it precedes the other three.

Then revise your register twice: first when automatic refunds are connected, then when private retrieval is removed from customer chat. A strong result changes the affected entries and preserves the ones whose paths remain. It does not copy the original ranking unchanged.

Go deeper

• Least privilege and defence in depth helps separate an enforcing boundary from an additional behavioral layer.
• Sensitive information disclosure expands the places private information can escape after a permitted read.
• Threat, vulnerability and risk helps you explain the register without mixing evidence and consequence.
• Denial of wallet adds budget and availability constraints when the assistant becomes publicly reachable.

Key takeaways

• Priorities follow the reachable design and its evidence, not attack-name popularity.
• Same-tenant privacy can fail even when tenant isolation works.
• Each control needs an owner, an enforced effect and a refusal test.
• New execution authority or removed data access should change the risk register.