AI Incident Response

TL;DR: Respond to the system’s actual reads, releases and state changes. Contain further harm while protecting relevant evidence, separate confirmed effects from hypotheses and unknowns, and investigate the failed permission or execution path. Recovery requires repaired controls and reconciled state, not just a new model, a rollback or a clean replay.

Before you read

Pick an answer before you read further. The lesson below explains it, and guessing first, right or wrong, helps the explanation stay with you.

1. 1An assistant reports that an external action was blocked, but the downstream service has not returned a final receipt. Can the incident record mark the action as safely denied?

   Commit to a guess, then reveal

What you should be able to do

Build an incident record that separates reports, verified events, causal hypotheses and unresolved effects. Choose containment that reaches the real execution boundary. Specify a protected evidence plan and a recovery decision that covers pending work, retained state and useful permitted tasks.

NIST SP 800-61 Revision 3, published in April 2025, integrates incident response with CSF 2.0 risk management. Preparation and improvement continue alongside detection, response and recovery. The workflow below applies that framing to an AI application; it is not a new mandatory NIST checklist.

Start from the reported effect

In this original fictional exercise, a delivery assistant reads account records, writes customer-visible notes and can change a pickup slot. A customer reports seeing a different account’s private delivery note. The assistant also says it changed a slot. The assets are customer information and the integrity of routing decisions; the relevant boundaries are account access, output readership and permission to change a booking.

Open an incident record and assign the response lead under the organization’s plan. A report deserves triage without being treated as a complete causal explanation. Preserve the reported artifact through an approved channel, identify its request and recipient, and check the relevant service records. Avoid asking the reporter to reproduce a disclosure involving another person’s real data.

An unauthorized read can be a separate incident even if the final answer never reveals the data. Conversely, a permitted internal artifact does not establish disclosure to a customer. Name the requirement and the reader at each boundary.

Contain without waiting for complete attribution

Choose the smallest control that can actually stop the suspected harm. If the path or affected population is uncertain, a broader restriction may be necessary immediately. The decision includes availability and business consequences, but a convenient narrow switch is insufficient if an uncontained path remains.

Disable the affected read, publication or action path in application enforcement. Check background workers, alternate credentials, queued jobs and retries. A model instruction to stop is not a kill switch. Draft-only operation can still expose information when customers, collaborators or downstream systems read the draft. Read-only operation can still disclose confidential records.

Preserve relevant state in parallel with urgent containment where feasible. Do not keep an active harmful path open merely to finish a snapshot. Record what changed, who authorized it, when it took effect and what evidence was lost. Credential revocation or rotation belongs to the credential owner’s response process; its effect depends on token type, caches and downstream acceptance.

A useful containment update says: “Customer-note publication is disabled at the publication service. Existing queued writes are held. Two already dispatched jobs remain under reconciliation.” It does not say “the agent is stopped” when only new sessions are blocked.

Collect evidence for specific questions

Begin with identifiers, provenance and effect records. Collect raw content only when it is needed, permitted and protected. Complete prompt logging is not a universal prerequisite to investigating a security incident, and a complete prompt does not reveal the model’s internal causal reasoning.

Restrict evidence access from collection onward. Separate the protected source record from redacted working extracts, and retain a transformation history. An incident-specific preservation request should have an authorized scope and review condition; calling every request a legal hold does not create legal authority. See sensitive-log evidence handling for the collection decision.

Investigate hypotheses, not a model confession

Map the reported data through session state, retrieval, tool responses, caches, stored outputs and any relevant training or fine-tuning lineage. Prioritize using the observed path and available evidence. There is no universal ordering that makes retrieval more likely than an ordinary application identity bug in every system.

A source preceding an action is evidence of opportunity, not proof that its wording caused the action. The model’s retrospective explanation is a claim to check. Controlled comparisons can test a hypothesis, but the intervention may also change context length, available facts or workflow behavior. Record those differences.

Use isolated systems you own or are explicitly authorized to test, synthetic accounts and controlled recipients after the permission checks under investigation. Do not replay an incident against live external recipients. Preserve relevant cache or queue conditions when they are part of the hypothesis rather than resetting away the suspected defect.

A successful replay can strengthen a mechanism explanation. Failure to reproduce does not invalidate a verified historical disclosure. Repeated trials describe the replay protocol; their frequency does not automatically estimate production incidence. Use the measurement lesson when making that additional claim.

Disabling retrieval also does not isolate training memorization by itself. Other runtime sources may remain, and similar text can be public, common or coincidental. Lack of a located source leaves uncertainty. It does not prove that the model invented the text, that a specific dataset contained it or that no privacy harm occurred.

Scope exposure and communicate what is known

Track source accounts, receiving principals, data categories, actions and the affected time window separately. Source-version matches, related configurations and shared faulty code help identify cases to examine. Keep confirmed, potentially affected and unresolved populations distinct. A missing log segment widens the uncertainty; it is not evidence that nobody else was affected.

Engineering supplies the event timeline, discovery and awareness times, impact evidence, limits and actions taken. The designated privacy, legal, contractual and communications owners determine applicable recipients and deadlines. Do not wait for perfect root-cause certainty before involving them.

For example, GDPR Articles 33 and 34 distinguish processor-to-controller notice, supervisory-authority notice and communication to affected people. Their conditions differ, and Article 33 allows information in phases. Check current applicability and the organization’s role; there is no single notification clock that applies to every AI incident.

Repair the boundary and reconcile its consequences

Remove or quarantine untrusted material through the authorized process after appropriate preservation. Follow its lineage into indexes, caches, memory, conversation state and derived datasets where relevant. Do not indiscriminately erase every store or assume that deleting the original document removes all useful or harmful derivatives.

Fix the failed access, publication or action control. If model artifacts or training data are implicated, involve their owners to verify provenance and choose a tested restoration or replacement. Retraining is not a default response to an application permission failure, and a matching artifact hash alone does not establish that its contents are acceptable.

Reconcile work already accepted by external services. A rollback cannot retract information already delivered, reverse every completed transaction or remove a retained artifact automatically. Compensation and customer remediation may remain necessary after the execution path is repaired.

NIST’s recovery guidance calls for checking restoration assets and restored operation against recovery criteria. For this assistant, require evidence of the repaired account boundary, permitted same-account work, held-job reconciliation and functioning observation before the authorized owner restarts the reviewed scope.

Produce a decision-ready incident record

Write a short update with six fields: verified event; affected and potentially affected scope; active containment; unresolved effects; next evidence check and owner; recovery condition. Link restricted evidence rather than copying private prompts into the update.

Then change one condition: the customer-visible note is repaired, but a background worker still uses the old identity mapping. The appropriate decision is to retain or extend the worker restriction, not to close the incident on the strength of the foreground fix. Explain which evidence would allow that restriction to be removed.

After recovery, turn confirmed failure mechanisms into regression cases, document remaining uncertainties and assign the logging or control improvements. Keep the original case history when later evidence corrects an early hypothesis. A useful review explains how the system permitted the event and how the response performed; it does not invent certainty or attribute intent without evidence.

Change the conditions

What if · interview follow-ups

Hold your answer as the constraints change

In an original fictional delivery-assistant incident, a customer receives another account’s private delivery note. A routing change is also proposed, but its final status is missing.

1. 1

   Start here

   What if the final answer says the routing change was refused?

   Nudge meAnswer it out loud, then reveal

Revising, not practising? Show every answer

Check your understanding

Check yourself

Say or write your answer first, then reveal. Explaining it from memory, in your own words, is the skill an interviewer is checking.

1. 1A source search finds twelve requests that retrieved a suspect document. Does this prove twelve customer disclosures?

   Answer it, then reveal

2. 2The team can stop an active leak immediately, but a full snapshot will take another hour. What should it do?

   Answer it, then reveal

3. 3Private-looking text reappears with retrieval disabled. Does that prove it was memorized during training?

   Answer it, then reveal

4. 4The team rolls back the application and all current tests pass. What remains before recovery?

   Answer it, then reveal

5. 5What is the difference between an incident update and a confirmed root-cause statement?

   Answer it, then reveal