Data Poisoning

TL;DR: Poisoning attacks the training phase: the attacker inserts or modifies data so the trained model misbehaves. Goals range from availability (degrade overall quality) to targeted errors on chosen inputs to backdoors that activate on a trigger. Entry points include scraped web content, user feedback and fine-tuning data, third-party datasets and retrieval corpora. Carlini and colleagues (2023) showed poisoning web-scale datasets is practical for low cost, and a 2025 study found around 250 poisoned documents sufficed to implant a simple backdoor across the model sizes it tested. Defend with provenance and snapshot integrity, curation and filtering, anomaly detection, evaluation, and least-privilege write access to data.

How poisoning differs from other attacks

Evasion changes inputs at prediction time to fool a fixed model. Poisoning changes the model itself, through its data, so every later prediction is affected. In CIA terms it is an integrity attack on the model.

Clean-label poisoning makes it worse: the poisoned samples carry correct-looking labels, so human review of labels does not catch them (Shafahi et al., 2018, "Poison Frogs!").

Where poison gets in

• Web-scale scraped data. Carlini et al. (2023), "Poisoning Web-Scale Training Datasets is Practical", described two attacks: buying expired domains that datasets reference by URL, so later downloads fetch attacker content (split-view poisoning), and timing malicious edits to Wikipedia just before snapshots are taken (frontrunning). They estimated that poisoning 0.01% of LAION-400M or COYO-700M would have cost about $60.
• Small amounts can suffice. In 2025, Anthropic, the UK AI Security Institute and the Alan Turing Institute reported that around 250 malicious documents were enough to implant a simple trigger-based backdoor in language models from 600 million to 13 billion parameters, even though the largest saw more than 20 times as much clean data, suggesting the needed number does not grow with model or dataset size. The caveats matter: the backdoor was a narrow one (a trigger phrase causing gibberish output, a denial-of-service behaviour), and whether the result holds for more complex behaviours and larger models is open.
• Fine-tuning and feedback data are small and highly influential, and often written by users.
• Retrieval corpora are not training data, but poisoning a RAG index steers outputs in the same way without retraining.

The arithmetic: count, not percentage

The intuition most people bring to poisoning is a share: an attacker must control some fraction of the data. The 2025 result is easier to remember once it is worked through.

For a defender, the practical conclusion is that dilution is not a control. A dataset ten times larger needs about the same number of planted documents, so what matters is who can place documents in it and how much any one source can contribute.

Poison that survives later training

A common hope is that later training washes poison out. Three results say not reliably.

• Through post-training. Zhang and colleagues (2024) poisoned 0.1% of the pretraining data of models from 600 million to 7 billion parameters, then applied supervised fine-tuning and preference training (DPO). Three of their four attack objectives persisted.
• Through human feedback. Rando and Tramèr (2023) showed that poisoning the preference data used in RLHF can plant a universal jailbreak trigger. They also found it significantly harder to plant than in classic supervised settings, which is useful nuance: feedback pipelines are a real target, but not an easy one.
• Through safety training. Anthropic's Sleeper Agents work found deliberately backdoored behaviour surviving supervised fine-tuning, RL and adversarial training. See backdoors.

Poisoning as a privacy weapon

Poisoning is usually described as an integrity attack, but it can also make privacy attacks stronger. Tramèr and colleagues' Truth Serum (2022) showed that poisoning under 0.1% of a training set made membership inference, attribute inference and data extraction attacks against particular targets one to two orders of magnitude more effective. The same idea is used defensively: Jagielski and colleagues (2020) used crafted poison samples to measure how much privacy differentially private training actually provides, rather than trusting the theoretical bound.

A poisoning-resistant ingestion pipeline

The defences table below lists the controls. Arranged as a pipeline, they look like staged promotion: data earns its way into training.

Defences

Treat datasets as supply chain artifacts: record them in an AI bill of materials so that when a dataset is found poisoned, you can find every model trained on it.

Investigating a suspected poisoned model

When a model starts behaving oddly after a training run, treat it as an incident with evidence to preserve.

1. Freeze the model artefacts, the training data snapshot and the pipeline logs.
2. Diff the dataset against the last good version by hash to list what was added, and from which sources.
3. Test causation by training controlled variants with and without the suspect slice and comparing behaviour on the trigger.
4. Check the pipeline itself: credentials and logs for unexpected writes to the data store. A poisoned dataset can be a supply chain compromise rather than a content problem.
5. Roll back to the last verified model and use the AI bill of materials to find every downstream model trained on the same data.

Misconceptions interviewers listen for

• "Bigger datasets dilute poison." The near-constant count result contradicts this for the backdoor tested.
• "Label review catches poisoning." Clean-label poisons carry correct labels.
• "Safety fine-tuning washes out pretraining poison." Poison at 0.1% persisted through SFT and DPO for most objectives tested.
• "The 250-document result means any behaviour is easy to plant." It was a narrow denial-of-service backdoor; harder behaviours may need more.
• "Poisoning is only an integrity problem." It can multiply the success of privacy attacks.

How this shows up in interviews

"How can training data be poisoned, and how would you detect it?" is a core adversarial ML question. Strong answers distinguish poisoning from evasion, give the goal types, name realistic entry points with the web-scale and small-sample results, and present defences honestly as layers with limits, ending with provenance and lineage for response.

Say it in 30 seconds: "Poisoning attacks the training phase: the attacker changes data so the trained model degrades, makes targeted mistakes or carries a backdoor trigger. Entry points are scraped web content, user feedback, third-party datasets and RAG corpora; research showed web-scale poisoning is cheap and a few hundred documents can implant a backdoor. I defend with pinned, hashed snapshots and provenance, restricted write access, curation and anomaly detection, influence-limiting training and evaluation, and lineage so I can find affected models when poison is found."