TL;DR: Contain the suspected disclosure path and preserve relevant evidence together. Trace the data from its owner to the principal and component that read it, then to each recipient or stored artifact. Repair the actual authorization or reuse boundary, reconcile retained state and verify the recovery scope; do not assume the model or retrieval layer is the cause.
How to approach it
Establish what the report shows and which confidentiality rule applies. Ask whether the data is verified, who owns it, who received it and whether that recipient had access. Treat a credible report seriously while separating confirmed disclosure from an unresolved resemblance or screenshot.
A strong answer
In an original fictional case, a procurement assistant returns a private supplier note from tenant B to a user in tenant A. The application shares retrieval and cache infrastructure. The protected asset is B’s note; an A user may supply queries but cannot authorize access to B’s records.
Contain the relevant path. Restrict the feature, source, publication route or reusable artifact as the evidence warrants. Broaden containment if the shared implementation leaves the affected population uncertain. Include workers and cached outputs. Preserve permitted state and logs in parallel; do not delay stopping active exposure for an exhaustive snapshot.
Follow the data, then test explanations. Use the report’s request and artifact IDs to join authenticated identity, delegation, source version, cache behavior, assembled context, service reads and actual recipients. An instruction not to disclose other tenants is not an access check.
Content enters the wrong context
Account B source
Private note
Account A context
Read is prohibited
Customer output
Trace actual readership
Outcome, unsafe: Upstream access failure
A later refusal does not erase the forbidden read.
A stored answer reaches the wrong reader
Authorized original run
B’s permitted request
Cached artifact
Reuse permissions matter
Account A receives it
Unauthorized reuse
Outcome, unsafe: Reuse boundary failure
Correct original retrieval does not authorize another reader.
Follow the artifact
Use identities, source versions, cache records and receiver evidence to locate the crossing. Do not choose a cause from the symptom alone.
| Hypothesis | Evidence that would strengthen it | What that evidence does not establish alone |
|---|---|---|
| Unauthorized retrieval | B’s restricted content entered A’s prohibited context through the read service | Whether another component also leaked it |
| Incorrect cache reuse | A received an artifact generated under B’s authorization context | Whether original retrieval was wrong |
| Session or memory crossing | A’s execution loaded state bound to a different principal | Whether the model memorized that state |
| Wrong tool identity | A’s action used an unintended credential or selected an unauthorized object | Whether the final output reached another reader |
| Training-related disclosure | Supported dataset lineage and controlled evidence after accounting for runtime sources | That every repeated private-looking string has that origin |
Do not rank these causes by an unsupported universal likelihood. Follow the application’s observed path. A shared index or prefix cache is not inherently a disclosure; the relevant issue is whether the implemented access and reuse rules permit an unauthorized observation.
Repair the permission boundary. Obtain tenant and user identity from the authenticated server context. Apply the required object and action checks before prohibited access or release, including reuse paths. OWASP’s authorization guidance recommends checking permissions on every request; successful authentication alone does not grant access to every resource.
A candidate-ranking step inside an authorized service can precede filtering without exposing content to the requester. Filtering after forbidden text has already entered the requester’s model context is too late for that access requirement. Explain the actual principal and observation point instead of arguing from the word “retrieval.”
Cache keys need the context that determines permitted reuse, or the release path must reauthorize against current permissions. Adding tenant ID alone does not address private records within a tenant or revoked access. Separate indexes may simplify one boundary but do not repair a wrong identity passed to a downstream tool.
Scope and reconcile. Record confirmed source owners, recipients, data categories and time windows. Keep potentially affected requests separate from confirmed disclosures and note collection gaps. Give the designated legal and privacy owners this evidence for notification decisions; engineering logs alone do not determine every obligation.
Quarantine or remove contaminated operational artifacts through the approved process after appropriate preservation. Check memory, caches, conversation history, generated reports and pending jobs. If training data or model artifacts are implicated, involve their owners to choose a validated replacement or remediation. Neither retraining every mixed-data model nor claiming that all data can be removed from weights is justified by the symptom alone.
Exercise the repair. In an isolated system you own or are authorized to test, use synthetic records to cover A-to-B denial, permitted B-to-B access, two users with different rights inside B, revocation after a warm cache hit, and background-job execution. Confirm the actual reader and the state at release. Retain a useful permitted task so a blanket outage cannot masquerade as a correct fix.
What interviewers probe next
- “It happened only once.” A confirmed disclosure remains an incident; frequency does not erase the requirement or decide notification applicability.
- “The no-retrieval run still prints the note.” Investigate remaining sources and provenance; that is not a unique test for memorization.
- “The regression suite passes.” Review coverage, retained state, observation health and restart criteria before restoring the scoped feature.
What if?
What if · interview follow-ups
Hold your answer as the constraints change
In an original fictional procurement assistant, a customer receives a private supplier note belonging to another tenant. Both tenants use the same application and shared retrieval infrastructure.
Start here
What if the search service ranks candidates before applying an access filter?
Locate the first prohibited exposure.
Inspect which principal and component may access those candidates and when unauthorized content becomes visible. Ranking inside an authorized service followed by correct filtering before disclosure is not automatically a tenant leak. A filter after private text has entered an unauthorized context or response is too late.
Carry it forwardThe enforcement boundary is the prohibited access, not a pipeline label.
Weak answerDeclaring every post-ranking filter a data breach.
Constraint changes
What if adding a tenant ID to the cache key fixes the reported case?
Are all users in that tenant equally authorized?
Retest object-level permissions, user-specific access, revocation, destination readership and warm-cache behavior. A tenant key can close one crossing while leaving private records visible to another user in the same tenant. Bind reuse to the applicable authorization context or reauthorize before releasing the artifact.
Carry it forwardTenant separation is one part of the permission model.
Weak answerTreating tenant membership as permission to read every object.
Constraint changes
What if the note appears after retrieval is disabled?
Which other sources remain?
Check conversation state, response caches, tool results and stored artifacts before attributing the text to training. Review training provenance where available, but persistence alone cannot identify memorization or rule out a runtime source. Keep the incident contained while the supported explanation develops.
Carry it forwardAn intervention narrows hypotheses only to the extent it isolates them.
Weak answerCalling a no-retrieval result proof of memorized customer data.
Now defend it
What evidence would justify restoring the feature?
Show the repaired access or reuse control on the original failure and adjacent permission cases, including allowed work and revocation. Reconcile retained artifacts and jobs, state the known and uncertain affected scope, verify monitoring and obtain the authorized scoped restart. Passing a few canaries does not prove the entire class absent.
How it's scored
Weak: Adds a prompt instruction or attributes the report to training without tracing the data.
Hire: Contains, identifies the crossing, scopes recipients and repairs access or reuse controls in the application.
Strong Hire: Distinguishes tenant and object permissions, reasons about filtering at the actual boundary, tests warm-cache revocation and allowed work, and reconciles stored artifacts before recovery.
Common mistakes
- Counting related source reads as confirmed customer disclosures.
- Assuming every shared component violates isolation.
- Deleting operational state before considering evidence requirements.
- Claiming a fixed case proves that every related path is closed.