AI Security Incidents: The Controls Around the Model

A model can refuse a harmful request while the database behind it exposes customer conversations. It can answer correctly while a cache returns the answer to the wrong person. Reviewing the model alone leaves both failures untouched.

The five public cases below show why an AI security review needs to follow the data through the whole application. They are selected examples, not evidence that one class of incident is more common than another. Read each case with one question in mind: which component had the authority to prevent the harm?

Five failures, five boundaries to inspect

1. A database exposed conversation logs

In January 2025, Wiz Research found a DeepSeek ClickHouse database accessible without authentication. It contained chat history, API secrets and operational information. Wiz reported that DeepSeek secured the exposure after disclosure. The report establishes an exposure; it does not establish who else accessed the data. Wiz's original investigation.

The boundary to inspect is access to the database. Restrict administrative interfaces to approved networks and identities, grant only the required database permissions, and keep secrets out of logs. Model refusal behavior cannot compensate for a direct database connection.

2. A job interface accepted work from outsiders

Oligo's March 2024 ShadowRay report described attackers abusing exposed Ray clusters to run unauthorized workloads and obtain credentials. The deployment assumptions mattered: the job interface was intended for a trusted environment, but the affected clusters were reachable by attackers. This describes the reported 2024 systems, not the authentication features of every current Ray release. Oligo's investigation.

Treat a job-submission endpoint as an execution boundary. Network restrictions and authenticated, authorized submission belong before the job runs. Worker identities also need narrow permissions, because a compromised worker should not inherit the whole cloud account. Practice the decision in exposed ML dashboards.

3. A client-library bug crossed user boundaries

On 20 March 2023, some ChatGPT users could see other users' conversation titles. OpenAI traced the incident to a redis-py bug involving cancelled requests and pooled connections. It also reported possible exposure of limited billing information for a subset of active Plus subscribers during a specific window. Full credit-card numbers were not exposed. OpenAI's incident account.

A cache key alone would not have fixed this response-mixup bug. Apply the library fix, verify that returned objects belong to the requesting identity, and test cancellation and concurrency across tenants. The cross-user data leak question gives you a response scenario to work through.

4. Loading weights also executed code

JFrog's February 2024 investigation described a malicious model whose pickle content executed code when loaded. That is a different failure from a model producing an unsafe answer: the loader executes the artifact before inference even begins. JFrog's original report.

Prefer data-only weight formats such as safetensors. That removes pickle's executable-deserialization mechanism, but accompanying Python code, loader vulnerabilities and malicious model behavior still need review. Hugging Face's safetensors audit account explains the format's purpose and the limits of an audit. Inspect the full loading path, not just the filename; see pickle versus safetensors.

5. The release pipeline published a malicious artifact

PyPI's December 2024 analysis of the Ultralytics compromise describes attacks through a GitHub Actions workflow and, subsequently, a publishing token. The first malicious releases came through the existing publishing workflow. Attestations helped investigators identify that route; their presence did not mean the package was benign. PyPI's incident analysis.

Protect build inputs and release credentials, isolate untrusted CI work from publishing, and verify provenance against an explicit policy. A pinned hash prevents an artifact from silently changing; it does not make an already malicious artifact acceptable. The build provenance lesson explains what the evidence can establish.

What changes when the model can act

These cases do not remove the need to study prompt injection. A retrieved document can influence an assistant's next action even though the document's author has no authority to request that action. An email tool, file search or browser can turn that influence into a disclosure or an unauthorized change.

Even a text-only assistant can disclose private material in its reply. Tools add more routes, including messages, uploads and network requests. The design task is to identify every route by which sensitive data or consequential actions can leave the boundary you intended.

A scoped credential limits what a tool can do after the model is misdirected. It does not prevent the model from being misdirected. That distinction keeps the control claim testable.

Try the review on one small system

Consider a hypothetical support assistant. It searches private customer tickets, caches answers by question text and can email a transcript. Each request has an authenticated tenant ID, but retrieval uses one service account for every tenant. The team proposes adding a prompt that tells the assistant to respect confidentiality.

Before reading on, write down the first two boundaries you would change and one test for each. Then decide whether the email tool belongs in the initial release.

Worked reasoning. First, enforce tenant and object permissions in the retrieval service using the server's authenticated identity. Test that a request from tenant A cannot retrieve a ticket belonging to tenant B, even when the requested identifier is valid. The model should never receive that unauthorized ticket.

Second, partition or disable the answer cache until it respects the same access rules. A matching question is not permission to reuse another user's answer. Test identical questions from different tenants, permission revocation and concurrent requests. Recheck authorization before serving cached private data; identity in the key alone does not handle a later permission change.

For email, ask what user need requires sending. If drafting is enough, keep the tool draft-only. If sending is required, validate recipients and permitted data in server code, then require review of the exact recipient, body and attachments. A familiar recipient address is not proof that the disclosure is appropriate.

What remains? Authorized users can still receive incorrect answers, and a user may approve a misleading draft. Record those residual risks and decide who owns the release decision. A prompt can improve behavior, but it cannot enforce the missing data-access rules.

Turn the review into evidence

For a system you own or are authorized to assess, record the asset, caller, operation and enforcement point for each boundary. Separate prevention from detection and containment. An authorization rejection prevents that operation; an alert helps detect it; a restricted worker identity limits the consequences of compromise.

Retain enough evidence to investigate: request IDs, caller identity, policy decisions, tool outcomes and references to the relevant source records. Apply access controls, redaction and a retention period. Copying every prompt, email and secret into a second log store creates another sensitive system to protect.

Use infrastructure security to practice exposed-service decisions, then agent security for actions taken with delegated authority. For each answer, name one control, the exact path it blocks and a path it leaves open.