Lesson 5 of 6 · 6 min read
Model Extraction
In this lesson
TL;DR: Model extraction turns API access into a copy of the model. Functionality stealing trains a substitute on the target's input-output pairs (for LLMs, distillation from generated responses). Parameter extraction recovers actual internals: Carlini and colleagues (2024) recovered the final embedding projection layer of production OpenAI models through their API using log-probability outputs. Attackers gain the model's value without its cost, a white-box stand-in for crafting transferable attacks, and insight into confidential details. Defend with per-key rate and volume limits, restricting output detail (labels instead of scores, limited or no log-probabilities), monitoring for systematic query patterns, terms enforcement, and watermarking where it applies.
Before you read
Pick an answer before you read further. The lesson below explains it, and guessing first, right or wrong, helps the explanation stay with you.
1Why would an attacker who only wants to bypass a content filter bother to extract a copy of the filter model?
Two kinds of theft
Functionality stealing. The attacker queries the target with many inputs, records outputs, and trains a substitute model to reproduce them. Tramèr and colleagues (2016), "Stealing Machine Learning Models via Prediction APIs", showed this against commercial machine-learning services, sometimes with few queries when confidence scores were returned. For language models, the equivalent is distillation: generating large volumes of responses from a stronger model to train a cheaper one. Many providers' terms prohibit using outputs to train competing models.
Parameter extraction. Sometimes specific internals can be recovered. Carlini et al. (2024), "Stealing Part of a Production Language Model", recovered the embedding projection layer, and with it the hidden dimension, of several production models through API features that exposed log-probabilities and logit biases. The providers changed their APIs in response.
The numbers behind the headlines
| Work | What was recovered | Cost or condition |
|---|---|---|
| Tramèr and colleagues, 2016 | Near-perfect copies of logistic regression, neural network and decision tree models hosted on BigML and Amazon Machine Learning | Few queries when confidence scores were returned; label-only outputs still allowed extraction, at higher cost |
| Carlini and colleagues, 2024 | The complete embedding projection matrix of OpenAI's Ada and Babbage models, confirming hidden dimensions of 1,024 and 2,048; the hidden dimension of gpt-3.5-turbo | Under $20 for each smaller model; an estimated under $2,000 for gpt-3.5-turbo's full matrix |
| Finlayson, Ren and Swayamdipta, 2024 | The hidden size of gpt-3.5-turbo, the ability to detect model updates and to identify which model produced an output | Under $1,000 of queries |
The 2024 results rest on a structural fact called the softmax bottleneck. A language model's last layer turns a hidden vector of d numbers into scores for every word in its vocabulary, tens of thousands of them. Because every score vector is produced from only d underlying numbers, all of them lie in a d-dimensional subspace. Collect enough full output vectors and their combined rank reveals d; push further and the projection itself can be recovered up to symmetries. The defensive lesson is specific: exposing log-probabilities together with arbitrary logit bias gives an attacker those output vectors cheaply. Restricting that combination, or adding noise, is what the providers changed.
What the attacker gains
| Gain | Why it matters |
|---|---|
| The model's capability without the training cost | Direct commercial loss |
| A white-box substitute | Easier, unlimited crafting of evasion inputs and jailbreaks that may transfer |
| Hidden details | Architecture size, training choices, confidential system behaviour |
| A model free of the provider's safeguards and terms | Misuse without monitoring |
What makes it easier
- Rich outputs: full probability vectors and log-probabilities carry far more information per query than a label or a single sampled text.
- High query volume: no per-key limits, or many keys.
- Deterministic, consistent responses: easier to learn from.
- Free tiers and trials: cheap access at scale.
Protecting: Model capability
- 1
1 · Per-key rate and volume limits
deterministic
- Stops: Millions of queries
- 2
2 · Limited output detail
deterministic
- Stops: Harvesting log-probabilities
- 3
3 · Query-pattern monitoring
detection
- Stops: Systematic input sweeps
- 4
4 · Account and payment verification
friction
- Stops: Many accounts in parallel
What each output feature gives away
- 1
1 · Label or single answer
Classifier decision, sampled text
- Least signal per query
- Extraction still possible with volume
Raises cost, does not prevent
- 2
2 · Many sampled responses
Volume of generated text
- Enough to distil behaviour
- Governed by terms of use
Functionality stealing
- 3
3 · Top-k log-probabilities
A few scores per position
- Richer training signal
- Useful to legitimate users too
Offer with limits and vetting
- 4
4 · Full probability vectors
Scores over the whole vocabulary
- Reveal the hidden dimension
- Fingerprint the model and its updates
Internals leak
- 5
5 · Log-probabilities plus logit bias
Attacker steers and reads scores
- Cheap recovery of the projection layer
- Changed by providers after disclosure
Avoid this combination
Low-detail outputs
Labels and sampled text still allow distillation, but each query teaches the attacker little.
Detecting extraction, and what looks like it
Extraction traffic tends to differ from product traffic in a few ways: sustained volume near limits; unusually broad coverage of the input space rather than the task-shaped clusters real users produce; for classifiers, inputs bunched near decision boundaries; heavy use of log-probability features; and many keys that share payment instruments, IP ranges or devices. The hard part is false positives. Benchmark runs, academic evaluations and some legitimate distillation within the terms look similar. Tiered access works better than blanket blocking: verified customers get richer outputs under contract, and anonymous or new accounts get less.
Worked example: pricing a distillation campaign
Suppose a competitor wants 10 million responses from your model to train their own. The numbers are illustrative.
| Your controls | What the attacker needs |
|---|---|
| No per-key limit | One key, a few days |
| 50,000 requests a day per key | 200 key-days: one key for 200 days, or 20 keys for 10 days |
| Per-key limit plus payment, IP and device correlation | 20 accounts with separate payment instruments and infrastructure, or detection |
Prevention is not the claim. The aim is to make the campaign slow, expensive and visible, so that terms can be enforced before it finishes. Removing the logprob and logit-bias combination separately addresses parameter extraction, which needs far fewer queries.
Limits per key only
20 keys, 50,000 requests a day each
Each key under its limit
nothing trips
10 million responses in 10 days
Outcome, unsafe: Campaign completes
Per-key limits alone are defeated by more keys.
Limits plus account correlation
20 keys, 50,000 requests a day each
Shared payment and IP ranges linked
one actor, many keys
Review and terms enforcement
Outcome, safe: Stopped part-way
Coverage patterns and shared infrastructure give the campaign away.
Sybil keys
An attacker who can open accounts simply divides the work.
Defences
- Rate and volume limits per key, per account and per payment identity, with alerts on sustained high-volume use.
- Limit output detail: return labels instead of scores for classifiers; restrict log-probability features to vetted customers or a small top-k; avoid exposing logit-bias features that enable parameter recovery.
- Monitor query distributions: extraction often looks like systematic coverage of the input space, unusual diversity, or synthetic-looking prompts at volume.
- Terms and enforcement: prohibit training competing models on outputs, and act on violations.
- Watermarking: statistical watermarks in generated text or in model behaviour can support later detection of a copy, with limits on robustness.
- Protect the real thing: extraction is slower and lossier than stealing weights, so weight protection remains the priority for the most capable models.
Distillation: terms, evidence and consequences
Not all distillation is theft. Teams routinely distil their own large models into smaller ones, and some providers allow it. Where terms prohibit using outputs to train competing models, enforcement needs evidence: output overlap on distinctive prompts, behavioural fingerprints, or watermarks. All of these have limits, since watermarks can be weakened by paraphrase and fingerprints by further training. After any confirmed case, ask what made extraction cheap and change that.
The security consequence is easy to miss. A good substitute is a white-box stand-in: an attacker can craft adversarial inputs and jailbreaks against it with gradients and unlimited queries, then use them against your model or your filters, relying on transfer.
Misconceptions interviewers listen for
- "Label-only outputs stop extraction." They raise the cost; Tramèr and colleagues extracted models without confidence scores.
- "Extraction means stealing all the weights." Partial internals, such as the hidden dimension and projection layer, are valuable and enable further attacks.
- "All distillation is theft." Terms and context decide; much distillation is legitimate.
- "Rate limits solve it." Attackers use many keys; link accounts by payment, network and behaviour.
What if · interview follow-ups
Hold your answer as the constraints change
You run security for a company that sells API access to its language model, including log-probabilities for some customers. The interviewer probes extraction risk and how you would respond.
Start here
What is the difference between functionality stealing and parameter extraction, with numbers?
Distillation versus recovering internals.
Functionality stealing trains a substitute on the model's outputs; for language models that is distillation, which needs volume. Parameter extraction recovers specific internals: in 2024, Carlini and colleagues recovered the full embedding projection of two production models for under $20 each and estimated under $2,000 for gpt-3.5-turbo, using log-probabilities together with logit bias.
Carry it forwardDistillation needs volume; parameter extraction needs revealing output features.
Weak answerTreating extraction as only full weight theft.
Constraint changes
How would you detect extraction on your API?
Volume, coverage, features, linked accounts.
Look for sustained volume near limits, input coverage that is broad rather than task-shaped, heavy use of log-probability features, and many keys that share payment, IP ranges or devices. Expect false positives from benchmark and research workloads, so route alerts to review and use tiered access instead of automatic blocking.
Carry it forwardCorrelate behaviour across keys, and plan for legitimate look-alikes.
Weak answerRelying on a per-key request threshold.
Constraint changes
A large customer wants full log-probabilities and logit bias. What do you offer?
Which combination made parameter recovery cheap?
Top-k log-probabilities without arbitrary logit bias in the same request, for verified customers under contract terms that forbid training competing models, with monitoring of their usage. Full vectors plus logit bias is the combination that let researchers recover internals cheaply, so it should not be available at all.
Carry it forwardOffer useful detail with limits; never expose bias plus full scores.
Weak answerGranting the full feature set because the customer is large.
Constraint changes
What if you learn a competitor has distilled your model?
Evidence, enforcement, then hardening.
Gather evidence such as output overlap on distinctive prompts, behavioural fingerprints and any watermark signals, knowing each can be weakened; hand it to legal for terms enforcement. Then find what made the campaign cheap, such as missing account correlation or generous free tiers, and fix that.
Carry it forwardEnforce with evidence, then remove what made extraction cheap.
Weak answerAssuming watermarks alone will prove the case.
Constraint changes
Why does a stolen substitute model matter for security, not only for revenue?
What can an attacker do with a white-box copy?
It gives the attacker a white-box surrogate: unlimited queries and gradients for crafting jailbreaks, filter evasions and adversarial inputs, which can then transfer to your production model and its classifiers. It may also reveal confidential behaviour such as system-level instructions learned in fine-tuning.
Carry it forwardA copied model is a practice range for attacks on the original.
Weak answerCalling extraction purely an intellectual property issue.
Now defend it
As the defender, what would you log to support an extraction investigation?
Per-key and per-account request volume over time, which output features were used, a sample or embedding summary of inputs to measure coverage, payment instruments, IP ranges and device fingerprints for linking accounts, and changes to tier or feature access.
How this shows up in interviews
"What is model extraction, what does the attacker gain, and how do you detect and limit it?" is a standard question for ML security and API platform roles. Strong answers distinguish functionality stealing from parameter extraction, cite a real result, connect extraction to transferable attacks, explain why rich outputs help attackers, and propose layered, realistic defences rather than claiming prevention.
Say it in 30 seconds: "Model extraction uses query access to steal a model: train a substitute on its outputs, which for LLMs is distillation, or recover internals, as Carlini's 2024 paper did with a production model's embedding projection via log-probabilities. The attacker gets the capability without the training cost and a white-box stand-in for crafting attacks that transfer. I defend with per-key rate and volume limits, less output detail like no or limited logprobs, query-pattern monitoring, terms enforcement and watermarking where it helps."
Check yourself
Say or write your answer first, then reveal. Explaining it from memory, in your own words, is the skill an interviewer is checking.
1Why does returning only the top label instead of full class probabilities make extraction harder?
2A customer asks for full log-probabilities for every token for their evaluation pipeline. How would you decide?
3What is the softmax bottleneck, and why did it make parameter recovery possible?
4Why are per-key rate limits not enough against distillation?