adversarial ml
AI security interview questions tagged adversarial ml, across every topic.
14 questions · 5 unlocked for you
Concepts behind "adversarial ml"
Explanations of the mechanisms these questions rely on.
Foundational
Adversarial Examples and EvasionAn adversarial example is an input deliberately perturbed, often imperceptibly, so that a model makes a wrong prediction. Evasion attacks use them at prediction time against a fixed model, with or without access to its internals, and they often transfer between models. Defences such as adversarial training and certified robustness help within limits, and many published defences have been broken by adaptive attacks. The same idea appears in language models as optimised adversarial suffixes. Adversarial ML & Privacy
Foundational
Data PoisoningData poisoning manipulates training, fine-tuning or retrieval data so a model behaves as the attacker wants: degrading accuracy, causing targeted mistakes, or planting a hidden trigger. Models trained on scraped web data, user feedback or third-party datasets are exposed because attackers can influence those sources. Research has shown that poisoning web-scale datasets is practical and that small numbers of documents can implant backdoors in large language models. Defences rest on data provenance, curation, anomaly detection and evaluation, and none is complete. Adversarial ML & Privacy
Foundational
Membership Inference AttacksA membership inference attack tries to determine whether a specific record was part of a model's training data, using the model's outputs. Models behave measurably differently on data they were trained on, especially when they overfit, and an attacker can exploit that gap. For sensitive datasets, membership itself is private information. Differential privacy is the principled defence; regularisation, limiting outputs and data minimisation reduce risk. Adversarial ML & Privacy
Foundational
Training-Data Memorisation and ExtractionLanguage models memorise part of their training data verbatim, and an attacker can sometimes make them emit it: personal data, secrets, copyrighted text. Memorisation grows with duplication, model size and prompt context, happens before overfitting, and survives alignment, which only hides it. Defences stack: deduplicate, scrub, train with differential privacy where it fits, filter outputs knowing approximate copies slip through, and measure with canaries and extraction tests. Fine-tuning on customer data is the highest-risk case. Adversarial ML & Privacy
Foundational
Model ExtractionModel extraction uses query access to a model to steal its functionality or parameters: training a substitute that imitates it, or recovering specific internals. It lets attackers avoid training costs, bypass usage terms, and build better attacks against the original. Rich outputs such as probabilities and log-probabilities make it easier. Defences combine rate limits, restricting output detail, monitoring for extraction-like query patterns, contractual terms and, where useful, watermarking. Adversarial ML & Privacy
Core
Backdoors and Trojaned ModelsA backdoored model behaves normally except when a specific trigger appears in its input, at which point it does what the attacker chose. Backdoors are implanted through poisoned training or fine-tuning data or by directly modifying weights, and they reach victims through shared pre-trained models and datasets. They are hard to find because testing rarely includes the trigger, and research has shown some backdoors survive later safety training. Provenance, controlled fine-tuning, evaluation and architectural containment matter more than detection. Adversarial ML & PrivacySign in