Adversarial Examples and Evasion

TL;DR: An adversarial example is an input altered by a small, deliberately chosen perturbation that makes a model err, while people see no meaningful change. Evasion attacks craft them at prediction time: white-box with the model's gradients (FGSM, PGD), black-box with queries, or by transfer from a substitute model. The core reason they exist is that models rely on features humans do not. Defences: adversarial training (raises robustness against the perturbations trained on), certified robustness (provable guarantees in small regions), and system-level controls. Evaluate defences against adaptive attackers; many have fallen to them. LLM jailbreaks found by optimisation are the same phenomenon in text.

What makes an example adversarial

In 2013, Szegedy and colleagues showed that image classifiers could be made to misclassify images by adding perturbations too small for people to notice. Goodfellow and colleagues (2014) explained the effect partly as a consequence of models behaving almost linearly in high-dimensional input spaces, and introduced the fast gradient sign method (FGSM): nudge every input feature a small step in the direction that increases the model's loss.

Three properties make this a security problem:

1. Small perturbations, large effects. The change is bounded (for example, each pixel moves by a few intensity levels), yet the prediction flips with high confidence.
2. Transferability. Examples crafted against one model often fool other models trained for the same task, so attackers do not need the target model.
3. Physical realisability. Researchers have produced adversarial patterns that survive printing and photography, such as stickers that cause road-sign misclassification (Eykholt et al., 2018).

The threat model has numbers

"Robust to adversarial examples" means nothing until three parameters are fixed.

• The norm measures how the perturbation is counted. The ℓ∞ norm caps the change to any single feature; ℓ2 caps the overall size of the change; ℓ0 caps how many features change at all.
• The budget, ε, is the maximum size under that norm. For small images a common research setting is ℓ∞ with ε = 8/255: each pixel value, on a 0 to 1 scale, may move by at most about 0.031.
• The goal is either untargeted (any wrong answer) or targeted (a specific wrong answer, which is harder and usually what an attacker wants).

Norms are not interchangeable, which is why a claim must name one. A 32 by 32 colour image has 3,072 values. An ℓ∞ budget of 8/255 lets every one of them move by 0.031, which adds up to an ℓ2 size of up to about 1.74. A defence certified at ℓ2 = 0.5 therefore says nothing about that ℓ∞ attacker, and the reverse holds too.

Attacker knowledge

PGD (Madry et al., 2017) repeats small gradient steps and projects back into the allowed perturbation budget, producing much stronger attacks than one FGSM step. It is the standard baseline for evaluating robustness.

A cautionary history: defences that only looked strong

The adversarial examples literature is the best-documented case in AI security of defences that passed their authors' tests and failed against a better attacker. Interviewers for ML security roles expect you to know it.

The common failure is gradient masking: a defence that makes gradients useless to the attacker's optimiser without removing the vulnerability. The model is still wrong near the input; the attacker's map is just broken. Attacks that approximate the gradient differently, or that do not need one, walk straight through.

Defences and their limits

• Adversarial training trains on adversarial examples generated during training (typically with PGD). It is the most reliable empirical defence, at a cost in clean accuracy and compute, and it protects mainly against the perturbation type and size it was trained on.
• Certified robustness provides a mathematical guarantee that no perturbation within a small radius changes the prediction, for example randomised smoothing (Cohen et al., 2019). Guarantees cover small radii and specific norms.
• Input transformations and detection (denoising, compression, detecting adversarial inputs) have repeatedly been broken. Athalye, Carlini and Wagner (2018) showed that many published defences relied on masking gradients and fell to adaptive attacks.
• System-level controls often matter more: limit query access, require multiple independent signals for high-stakes decisions, keep humans in the loop for consequential outcomes, and monitor for probing.

The evaluation rule from this literature is worth remembering everywhere in AI security: a defence is only as good as its performance against an adaptive attacker who knows the defence.

Certified robustness, with a real number

Certification gives a proof rather than a test result: for a given input, no perturbation within radius r under a stated norm can change the prediction. The best-known scalable method, randomised smoothing (Cohen, Rosenfeld and Kolter, 2019), certified 49% top-1 accuracy on ImageNet against ℓ2 perturbations up to 0.5.

That radius sounds generous until it is spread across an image. A 224 by 224 colour image has 150,528 values; an ℓ2 budget of 0.5 spread evenly moves each one by about a third of one intensity level out of 255. Certification is real and valuable, but at practical scales it covers small perturbations under one norm, and accuracy under certification is far below a normal model's.

Robustness has costs

• Accuracy. Tsipras and colleagues (2018) showed that robustness can be provably at odds with standard accuracy for some data distributions, matching the clean-accuracy drop seen in practice.
• Why it happens. Ilyas and colleagues (2019) argued that adversarial examples exploit non-robust features: patterns in the data that genuinely predict the label but are brittle and imperceptible to people. A model that uses every predictive pattern will use those too.
• Privacy. Song, Shokri and Mittal (2019) measured six adversarial-training defences and found they increased membership inference risk: hardened models can reveal more about whether a record was in the training set. A hardening project should include a membership inference audit.

Worked example: reading a robustness claim

A vendor says its image classifier for content moderation is "92% robust". Before that number means anything, ask:

Without answers, "92% robust" is a marketing figure. With them, it becomes a claim you can compare and test, and for a moderation system you would still pair it with rate limits on probing, human review of borderline cases and monitoring for evasion patterns.

Evasion in language models

Text is discrete, which makes gradient attacks harder, but the idea carries over. Zou and colleagues (2023) used gradient-guided search to find adversarial suffixes, strings appended to a prompt that caused aligned models to comply with requests they would refuse, and some suffixes transferred across models. Jailbreaks and prompt injection that slip past classifiers are close relatives of evasion: the attacker searches for inputs on the wrong side of a learned boundary. That is why filters cannot close the gap on their own.

NIST's taxonomy of adversarial machine learning (NIST AI 100-2, 2025 edition) classifies evasion alongside poisoning, privacy and abuse attacks and is a useful reference for vocabulary.

Misconceptions interviewers listen for

• "Keeping the weights secret prevents evasion." Transfer and query-based attacks work without them.
• "It resists PGD, so it is robust." Gradient masking fooled reviewers repeatedly; demand adaptive attacks.
• "Adversarial training fixes it." It helps within the trained threat model, costs clean accuracy and can raise privacy leakage.
• "Certified means secure." Only within a small radius under one norm.

How this shows up in interviews

"What is an adversarial example, and how is it different from an ordinary error?" opens many ML security interviews. Strong answers say it is an input chosen by an adversary to cause a specific error, contrast it with natural errors on the data distribution, explain white-box versus black-box and transferability, and describe adversarial training and certified robustness honestly, including the adaptive-attack lesson. Connecting it to LLM jailbreaks shows breadth.

Say it in 30 seconds: "An adversarial example is an input with a small, deliberately chosen perturbation that makes a model err while looking unchanged to people. It's evasion: the model is fixed and the attacker works at prediction time, with gradients like FGSM or PGD in white-box, queries in black-box, or transfer from a substitute model. Adversarial training and certified robustness help within limits, many detection defences have been broken by adaptive attacks, and optimised jailbreak suffixes are the same idea in LLMs."