Membership Inference Attacks

TL;DR: Membership inference asks whether a given record was in the training set. Models tend to be more confident and lower-loss on their training data, so an attacker who compares a record's behaviour against what is typical can infer membership. Overfitting widens the gap. The privacy harm is that membership itself can reveal sensitive facts: being in a hospital's diagnosis model means having the condition. The principled defence is differential privacy in training; practical mitigations are regularisation, returning labels rather than full confidence scores, and keeping sensitive data out of training. For large language models, measured attacks are often weaker than for small overfit models, but verbatim memorisation is a related and real risk.

Why models leak membership

Training pushes a model to fit its training examples. Unless generalisation is perfect, the model ends up slightly better on examples it has seen than on similar ones it has not: lower loss, higher confidence in the correct label. That difference is a signal.

Shokri and colleagues (2017) demonstrated the attack against machine-learning-as-a-service APIs using shadow models: train models on data from a similar distribution where you know membership, learn what members' outputs look like, and apply that classifier to the target. Later work, notably Carlini and colleagues' likelihood-ratio attack (LiRA, 2022), made attacks much stronger and argued they should be evaluated at low false-positive rates, because confidently identifying a few members matters more for privacy than average accuracy.

How LiRA works, and why the metric changed

The likelihood-ratio attack replaces one global threshold with a per-example test. The attacker trains many shadow models (256 in the paper's main experiments, 64 for ImageNet), some with the target example and some without. For that one example, this gives two distributions of the model's confidence: what it looks like when the example was a member, and when it was not. Each is fitted with a Gaussian, and the target model's confidence on the example is scored by which distribution explains it better. Some examples are naturally easy and some hard, and the per-example comparison removes that noise.

The paper's second contribution was the metric. Average accuracy or AUC rewards an attack that is slightly better than chance on everyone. Privacy harm comes from identifying a few people with high confidence, so the paper reports the true-positive rate at a very low false-positive rate, such as 0.1% or 0.001%. On that measure, the older loss-threshold attack achieved 0% true positives at 0.1% false positives, while LiRA identified between 0.1% and 10% of members at 0.001% false positives, depending on the dataset: about ten times more powerful at the low end.

Worked example: why accuracy hides the harm

An audit uses 10,000 training members and 10,000 non-members. The numbers are illustrative.

Attack A looks better by accuracy, but it cannot confidently identify anyone. Attack B identifies 300 people with 97% precision. If membership reveals that someone was in, say, a clinical study, B is the privacy problem.

What makes it worse

• Overfitting: small datasets, many training epochs, very large models relative to data.
• Outliers and duplicates: unusual or repeated records are memorised more and are easier to detect.
• Rich outputs: full probability vectors or log-probabilities give more signal than labels.
• Repeated queries: the attacker can average over many queries or augmentations.

Large language models

For large language models trained on huge corpora for few epochs, studies have found that many published membership inference methods perform close to chance in realistic settings (for example Duan et al., 2024), partly because members and non-members are hard to separate cleanly. That does not make LLMs private. Verbatim memorisation is the more pressing risk: models can reproduce rare or repeated training text, including personal data, when prompted, which is training data extraction. Fine-tuned models trained on small, sensitive datasets are closer to the classic setting and more exposed.

The distribution-shift trap

Many early papers reported strong membership inference against large language models. Duan and colleagues (2024) tested attacks on models from 160 million to 12 billion parameters trained on the Pile and found they barely beat random guessing. Two reasons: huge datasets seen for about one epoch leave little per-example signal, and the boundary between "member" and "non-member" text is fuzzy when near-duplicates are everywhere. Several apparent successes were explained by distribution shift: the non-member test set was drawn from a later time period, so the attack was detecting newer text rather than membership. An honest audit draws members and non-members from the same domain, time range and preprocessing, and reports a random-guess baseline next to the attack.

Defences

Measure risk empirically too: run membership inference audits against your own model before release, evaluated at low false-positive rates.

Differential privacy in practice

DP-SGD (Abadi and colleagues, 2016) is the standard way to train with differential privacy. Each training step clips every example's gradient to a maximum size, so no single record can move the model much, then adds Gaussian noise to the sum. A privacy accountant tracks how the guarantee degrades over many steps.

The guarantee is stated as (ε, δ). Informally, an attacker's ability to tell whether any one record was used is bounded: at a false-positive rate α, the true-positive rate can be at most about e^ε × α + δ.

Large ε values are common in practice because accuracy suffers at small ones. Abadi and colleagues reported 90%, 95% and 97% accuracy on MNIST at ε of 0.5, 2 and 8 (δ = 10⁻⁵), against 98.3% without privacy; on CIFAR-10 they reported 67%, 70% and 73% at ε of 2, 4 and 8, against about 80% without. The theoretical worst case at ε = 8 is weak, though measured attacks usually do far worse than the worst case.

Three further points separate experienced answers:

• Audit the implementation. Bugs in clipping, sampling or accounting break guarantees silently. Researchers audit DP training by inserting crafted canary records and measuring how detectable they are, which gives an empirical lower bound on the real ε.
• ε is per record. A person who appears in many records, or a record duplicated many times, is protected less. Deduplicate and consider user-level accounting.
• DP is not legal compliance on its own. The European Data Protection Board's Opinion 28/2024 says whether a model is anonymous is assessed case by case, looking at how likely extraction and inference are. DP is strong evidence for that assessment, not a substitute for it.

Also note a trade-off from the evasion literature: Song, Shokri and Mittal (2019) found that adversarial training increased membership inference risk. Hardening a model against evasion can make it leak more about its training data.

Misconceptions interviewers listen for

• "Accuracy or AUC is the right metric." Use true-positive rate at low false-positive rates, ideally on a log-scale ROC curve.
• "Membership inference works well on large pretrained LLMs." Measured attacks are mostly near chance there; check for distribution shift. Fine-tunes on small sensitive sets are a different story.
• "Differential privacy makes us GDPR compliant." It strengthens an anonymity assessment; it does not replace one.
• "Our configuration says ε = 8, so we are private." Audit the implementation, and remember ε is per record and weak at that size.
• "Robustness and privacy point the same way." Adversarial training has been measured to increase membership leakage.

How this shows up in interviews

"What is membership inference, and why does overfitting raise the risk?" is a standard ML privacy question. Strong answers explain the train/test behaviour gap, give the shadow-model idea, explain why membership is sensitive with a concrete example, and present differential privacy as the principled fix with practical mitigations alongside. Mentioning that LLM membership inference is often weak while memorisation is strong shows current knowledge.

Say it in 30 seconds: "Membership inference asks whether a specific record was in the training data. Models are more confident on what they were trained on, and overfitting widens that gap, so comparing a record's confidence or loss against shadow-model baselines reveals membership. That matters when membership itself is sensitive, like a disease-specific dataset. Differential privacy bounds it; regularisation, label-only outputs, deduplication and minimisation help. For big LLMs measured MIAs are often weak, but verbatim memorisation is a real risk."