Training-Data Memorisation and Extraction

TL;DR: Language models memorise part of their training data word for word, and an attacker can sometimes make them emit it: phone numbers, keys, private messages, copyrighted text. Memorisation grows with how often a sequence is duplicated, with model size and with how much context the prompt supplies; it happens before overfitting, and alignment hides it rather than removing it. In 2023, researchers spent about USD 200 to extract over 10,000 unique memorised examples from ChatGPT. Defences stack: deduplicate, scrub personal data, train with differential privacy where the cost is acceptable, filter outputs knowing that approximate copies slip past verbatim filters, and measure with canaries and extraction tests that use your real training recipe. Fine-tuning on customer data is the riskiest case: small, repeated, sensitive data seen late in training.

Before you read

Pick an answer before you read further. The lesson below explains it, and guessing first, right or wrong, helps the explanation stay with you.

1. 1A team fine-tunes a model on three epochs of customer support tickets. A membership inference audit comes back near chance, so they declare the model private. Why is that conclusion unsafe?

   Commit to a guess, then reveal

Three different questions

Keep three attacks apart. They are often confused in interviews.

• Membership inference: was this record in the training data? A yes-or-no question about one known record. See membership inference attacks.
• Training-data extraction: can someone make the model output its training data? This lesson.
• Model extraction: can someone copy the model through its API? See model extraction.

And keep memorisation apart from context leakage. A model that reveals its system prompt, a retrieved document or another user's cached answer is leaking runtime context, which is an access-control and design problem covered in sensitive information disclosure. Memorisation is about what is stored in the weights. In triage, rule out context sources first, because they are far more common.

What "memorised" means precisely

Researchers use several definitions, and the choice changes the numbers.

• Extractable (Carlini and colleagues, 2021): a string is extractable if some prompt makes it the model's most likely continuation.
• k-eidetic: extractable, and present in at most k training documents. A secret seen once and still extractable (k = 1) is the most worrying case.
• Extractable with k tokens of context: take a training sequence, give the model its first k tokens, and check whether it produces the rest exactly. This prefix-suffix test is what every vendor report now uses, typically with 50-token prefixes and 50-token continuations.
• Discoverable versus extractable (Nasr and colleagues, 2023): discoverable memorisation is measured by someone holding the training data, using true prefixes; extractable memorisation is what an attacker without the training data can get. On one 6-billion-parameter model they found 30.1% of tested examples discoverably memorised and 14.5% extractably memorised. Their advice to practitioners: test for discoverable memorisation, because it is far easier to measure and much of it becomes extractable with effort.
• Approximate memorisation: near-copies. Google's reports count a continuation within 10% edit distance; Ippolito and colleagues used a similarity threshold. Approximate memorisation is consistently larger than exact: about 14 times exact in the Gemini 1.5 report and about 24 times in Gemma 3.

What drives memorisation

Research since 2021 gives clear, quantitative trends.

• Duplication matters most. Kandpal and colleagues (ICML 2022) found regeneration is superlinear in duplicate count: a sequence present 10 times is generated about 1,000 times more often than one present once. In GPT-2, every URL inserted 33 or more times in one document was memorised by the largest model.
• Bigger models memorise more. Carlini and colleagues ("Quantifying Memorization", ICLR 2023) found a log-linear trend: a tenfold increase in model size added about 19 percentage points of memorisation on their duplication-heavy sample, and estimated that at least 1% of the Pile training set was extractable from a 6-billion-parameter model. Llama 3's report shows the same within one family: 0.24% for 8B, 0.55% for 70B and 1.03% for 405B on its 50-gram test. Recipes matter too: Gemma 3 found only marginal differences between its 4B, 12B and 27B models.
• More context reveals more. The same 6B model reproduced 33% of test sequences given 50 tokens of context and 65% given 450.
• Repeated epochs increase it, which is why fine-tuning for several epochs on a small dataset is risky. Examples seen early in pre-training are often forgotten by the end; examples seen late, such as fine-tuning data, do not get that benefit.
• It is not overfitting. The Secret Sharer study (Carlini and colleagues, 2019) found memorisation happens early, while the model is still learning, and that weight decay, dropout and quantisation did not stop an inserted secret being extracted.
• Unique, high-entropy strings can be memorised from one document if repeated inside it: GPT-2 reproduced random-looking identifiers that appeared in exactly one training document.

The landmark results

The ChatGPT result is the one to understand. Chat alignment made the model look about 50 times less memorising than base models. The researchers found that asking it to repeat a single word many times eventually made it diverge from chat behaviour and continue like a base model, sometimes emitting pre-training text; in that state it emitted memorised data about 150 times more often than normal. In 16.9% of tested generations the output contained memorised personal data. The authors disclosed to OpenAI in August 2023 and published in November; ChatGPT then began warning that such requests may violate its terms. The authors' framing is the lesson: refusing the trick patches the exploit, not the vulnerability.

Vendors now report memorisation in model cards. Google's Gemini 1.5 report re-ran the divergence test: 0.35% of diverged generations were training data, against about 2.8% for ChatGPT 3.5 in the original study. Gemma 2's report found no high-severity personal data in memorised output, and Gemma 3's found no personal information at all in its memorisation tests.

At about USD 200 for over 10,000 examples, extraction cost roughly USD 0.02 per memorised example. That is why rate limits and anomaly detection belong in the defence list.

Measuring it with canaries

The Secret Sharer method is the standard way to test your own training pipeline.

1. Build canaries: random secrets in a known format, such as "the account number is" followed by 9 random digits. The space of possible values R has 10^9 members.
2. Insert each canary into the training data a chosen number of times: 1, 2, 4, 8 and so on.
3. Train with the exact production recipe.
4. Rank each inserted canary among all candidates by how likely the model finds it. Exposure = log2 |R| − log2 rank.

Worked example: reading exposure

For a 9-digit canary, log2(10^9) ≈ 29.9, the maximum exposure.

The paper's validation found that, for a space of 2^30 candidates, extraction always succeeded above exposure 33 and never below 31 (estimated exposure can exceed log2 |R|). Plot exposure against insertion count for each recipe change; a jump tells you which change made memorisation worse.

Defences, and where each one stops

Deduplication. Lee and colleagues (ACL 2022) found models trained on deduplicated data emitted memorised text about 10 times less often, with the same or better accuracy; one 61-word sentence had appeared over 60,000 times in the C4 dataset. Limits: in the 2023 study, deduplication cut memorisation threefold for sequences repeated fewer than 35 times but did not help for sequences repeated about 400 times or more, because deduplication at scale is imperfect, and document-level deduplication misses repeats inside one document.

Scrubbing personal data and secrets. A first line of defence, never complete: unknown formats and typos slip through, and Lukas and colleagues (IEEE S&P 2023) found scrubbing reduced but did not eliminate personal-data leakage from fine-tuned models.

Differential privacy. DP-SGD bounds how much any one privacy unit can affect the model. Google's VaultGemma (2025), a 1-billion-parameter model trained from scratch with sequence-level DP at ε ≤ 2.0 for 1,024-token sequences, showed no detectable memorisation in Google's test. The price was capability: on the TriviaQA benchmark it scored 11.24 against 39.75 for the non-private Gemma 3 1B, comparable in Google's words to non-private models from about five years earlier. Two caveats: the guarantee is per sequence, so text repeated across many sequences can still be learned, and Lukas found sentence-level DP still leaked about 3% of personal-data sequences because one person's data spans many sentences. Match the privacy unit to what you are protecting, often the user. DP fine-tuning is much cheaper than DP pre-training and is the more practical option for customer data.

Output filters. GitHub Copilot compares suggestions plus about 150 characters of surrounding code against an index of public GitHub code, and finds matches in under 1% of inline suggestions. Ippolito and colleagues showed the general limit: a filter that blocks every verbatim copy still leaks near-copies, for example with changed variable names, and gives "a false sense of privacy". Filters also cover only what is in their index.

Alignment and refusals reduce casual emission, but Nasr's paper concludes current alignment does not eliminate memorisation.

Unlearning is not deletion yet. On the TOFU benchmark, none of the baseline methods unlearned effectively. Zhang and colleagues (ICLR 2025) found unlearned models kept about 21% of the targeted knowledge, rising to about 83% after 4-bit quantisation. Other work restored unlearned content by fine-tuning on loosely related public data.

Volume controls. Extraction at scale needs many queries, so per-key quotas, anomaly detection on repetitive or degenerate outputs, and terms against programmatic extraction all raise the cost.

Worked example: what ε means for an attacker

A common way to read a differential-privacy budget: if an attacker starts with prior belief p that a record was used, then (ignoring δ) their belief after any analysis of the model is at most e^ε·p / (e^ε·p + 1 − p).

At ε = 2 the bound is meaningful. At ε = 8 the formal worst case says little, although measured leakage can still be low; the Secret Sharer found canary exposure stayed at 3.2 or below for DP models even with very loose formal budgets, against 31 without DP. That is why DP deployments are also audited: Steinke and colleagues' one-run audit (NeurIPS 2023) inserts canaries to measure an empirical lower bound on ε, and a lower bound above the claimed ε reveals an implementation bug.

Fine-tuning on customer data: a test plan

Fine-tuning data is small, often repeated, sensitive, and seen for several epochs at the end of training, which is every risk factor at once. A practical plan, assembled from the methods above:

1. Clean the data. Remove exact and near-duplicates within and across customers; scan for personal data and secrets; drop high-risk fields.
2. Plant canaries in the formats of your real secrets at several insertion counts, and train with the exact production recipe.
3. Run prefix-suffix tests on a stratified sample of real records, by duplicate count, customer and field type; report exact and approximate rates.
4. Sample, not only greedy. Single greedy attempts underestimate what repeated sampling extracts.
5. Test through the deployed interface as well, because alignment and system prompts hide memorisation that base-style continuation reveals.
6. Isolate tenants. Per-customer fine-tunes or adapters mean memorised data can only reach the customer it came from. Prefer retrieval with authorization for facts that change or may need deletion.
7. Gate releases. No industry-standard thresholds exist, so set your own and say so. A reasonable starting point: no canary near maximum exposure at duplication levels the real data reaches, and no exact extraction of high-severity personal data or secrets, retested on every retrain.

Why lawyers now care

Memorisation measurements have become legal evidence.

• Data protection. The European Data Protection Board's Opinion 28/2024 says AI models trained on personal data cannot always be considered anonymous: anonymity requires that extracting personal data, directly or through queries, be insignificantly likely. It lists testing for regurgitation of training data among the evidence controllers should provide. That makes extraction testing a compliance activity, not just research.
• Erasure requests. If a person's data is memorised, the options are retraining without it, filtering outputs, or unlearning with the limits above. Per-tenant adapters make retraining cheap.
• Copyright. In November 2025 the Munich Regional Court held, in GEMA v OpenAI, that memorising song lyrics in model parameters was a reproduction; the judgment was not final when reported. In the same month, the UK High Court in Getty Images v Stability AI found no secondary infringement, with both sides' experts agreeing the image model's weights contained no copies of the works. In the New York Times case against OpenAI and Microsoft, the complaint's exhibit of 100 memorised articles started a fight over definitions, exact versus approximate and realistic versus engineered prompts, that mirrors the research literature; as of November 2026 the case had no summary-judgment ruling.

Misconceptions interviewers listen for

• "Memorisation is overfitting." It happens early, and regularisation did not stop it.
• "Membership inference near chance means the model is private." Large pre-trained models can show near-chance membership inference and still have at least 1% of training data extractable.
• "Aligned chat models do not regurgitate." Alignment hid ChatGPT's memorisation until a divergence attack raised emission 150 times.
• "A verbatim filter solves it." Approximate memorisation is many times larger than exact.
• "Deduplication solves it." It helps most for moderately repeated text and fails for heavily repeated text at scale.
• "DP at some ε means nothing leaks." The guarantee is per privacy unit; repeated content across units can still be learned.
• "Unlearning deletes the data." Quantisation and light fine-tuning have restored supposedly unlearned content.

What if · interview follow-ups

Hold your answer as the constraints change

Your company will fine-tune a model on customer support tickets and offer it to each customer. Legal asks whether the model can leak customer data, and the interviewer probes your answer.

1. 1

   Start here

   Why is fine-tuning on customer tickets riskier than the base model's pre-training?

   Nudge meAnswer it out loud, then reveal

Revising, not practising? Show every answer

Primary references

References for the mechanisms discussed above, accessed 6 November 2026:

• Carlini et al., "The Secret Sharer", 2019.
• Carlini et al., "Extracting Training Data from Large Language Models", USENIX Security 2021.
• Carlini et al., "Quantifying Memorization Across Neural Language Models", ICLR 2023.
• Nasr et al., "Scalable Extraction of Training Data from (Production) Language Models", 2023.
• Kandpal et al., "Deduplicating Training Data Mitigates Privacy Risks in Language Models" and Lee et al., "Deduplicating Training Data Makes Language Models Better".
• Ippolito et al., "Preventing Generation of Verbatim Memorization in Language Models Gives a False Sense of Privacy".
• Lukas et al., "Analyzing Leakage of Personally Identifiable Information in Language Models", IEEE S&P 2023.
• VaultGemma technical report and Steinke et al., privacy auditing with one training run.
• Memorisation sections of the Gemini 1.5, Gemma 3 and Llama 3 reports.
• TOFU unlearning benchmark and Zhang et al. on quantisation restoring unlearned knowledge.
• GitHub Copilot code referencing.
• EDPB Opinion 28/2024 on AI models.

How this shows up in interviews

Expect "can an LLM leak its training data?", "how would you test a fine-tuned model for memorisation?", "what is the difference between membership inference and extraction?", "does differential privacy fix it?" and "how do you handle a deletion request for data in a model?". Strong answers name what drives memorisation with numbers, explain canaries and exposure, distinguish discoverable from extractable, know the limits of filters, deduplication and unlearning, and treat fine-tuning on customer data as the high-risk case.

Say it in 30 seconds: "Models memorise some training data verbatim, more with duplication, model size and prompt context, and alignment hides it rather than removing it: researchers extracted over 10,000 examples from ChatGPT for about USD 200. I deduplicate and scrub, use DP fine-tuning when the privacy unit and quality cost fit, add output filters knowing near-copies pass, and measure with canaries and prefix-suffix tests on the exact production recipe. For customer data I fine-tune per tenant, so leakage stays within the tenant and deletion means retraining an adapter."

Check yourself

Say or write your answer first, then reveal. Explaining it from memory, in your own words, is the skill an interviewer is checking.

1. 1What is the difference between discoverable and extractable memorisation?

   Answer it, then reveal

2. 2How is canary exposure calculated?

   Answer it, then reveal

3. 3Name three factors that increase memorisation.

   Answer it, then reveal

4. 4Why do verbatim output filters give a false sense of privacy?

   Answer it, then reveal

5. 5Why is unlearning not a reliable way to delete memorised data?

   Answer it, then reveal