TL;DR: Model extraction uses queries to steal a model's functionality (train a substitute on its outputs, or distil an LLM from its responses) or its parameters (Carlini et al., 2024, recovered a production LLM's embedding projection layer through log-probability features). The attacker gains the capability without the training cost, a white-box substitute for crafting transferable attacks, hidden details, and use free of the provider's safeguards. Limit it with per-key rate and volume caps, less output detail, query-pattern monitoring, account verification, terms enforcement and watermarking; none prevents it outright.
How to approach it
Define both forms with a cited example, list the attacker's gains (including the less obvious one: better attacks), then present layered controls and how you would detect an extraction campaign in practice.
A strong answer
Attacker
many accounts
Model API
outputs, log probs
Query/response set
covers behaviour
Substitute model
white-box proxy
Detect and limit
Fewer outputs exposed
- Labels not full log probs where possible
Usage shape
- Coverage-style prompts, account linkage
Watermarks and terms
- Evidence for enforcement
Forms.
- Functionality stealing: query broadly, collect input-output pairs, train a substitute. Tramèr et al. (2016) did this against commercial prediction APIs; LLM distillation from a stronger model's responses is the modern form.
- Parameter extraction: exploit API features to recover internals. The 2024 production-model result relied on log-probabilities and logit bias; providers changed their APIs afterwards.
Gains.
| Gain | Consequence |
|---|---|
| Capability without training cost | Commercial loss |
| White-box substitute | Unlimited offline search for jailbreaks and evasions that may transfer back |
| Confidential details | Model size, behaviours, system design |
| Unmonitored use | Misuse outside the provider's safeguards and terms |
Limiting.
- Per-key, per-account and per-payment-identity rate and volume limits, with alerts on sustained high volume.
- Reduce information per query: labels rather than scores for classifiers; restricted or top-k log-probabilities for LLMs; no feature combinations known to leak parameters.
- Verification and friction for high-volume access; detect many accounts tied to one payer or network.
- Terms prohibiting training on outputs, with enforcement.
- Watermarking of outputs or model behaviour to support detection of copies, acknowledging limited robustness.
Detecting. Extraction campaigns tend to show high volume, broad and systematic coverage of the input space, synthetic or templated prompts, little conversational structure, and distribution unlike normal product use. Build per-key baselines and alert on outliers; review keys with sudden growth.
Perspective. Extraction yields an approximation; stealing the weights yields the model. Prioritise weight protection, and treat extraction as an API abuse problem with business and security dimensions.
What interviewers probe next
- "Can you detect distillation from normal usage?" Sometimes from volume and prompt patterns; a careful attacker spreading queries across accounts is hard to see, which is why terms and verification matter.
- "Should you add noise to outputs?" It can reduce information per query, but degrades quality for legitimate users; prefer restricting detail.
- "Is extraction a threat to an internal-only model?" Less so, but insiders and compromised accounts can still query at scale; keep limits and logging.
What if?
What if · interview follow-ups
Hold your answer as the constraints change
A competitor queries your paid image-classification API millions of times through many accounts, collecting full probability outputs for diverse synthetic inputs.
Start here
What is the competitor trying to get?
What can they do with a model that mimics yours?
A copy of the model's functionality: by training a substitute on your outputs, they gain the capability without your data or training cost. A substitute also gives them a white-box model for crafting transferable evasion attacks, and lets them use it free of your terms and safeguards.
Carry it forwardExtraction steals capability and enables further attacks.
Weak answerTreating it as ordinary heavy usage.
Constraint changes
Why do probability outputs help them?
How much does a label tell you compared with a probability vector?
Full probability vectors reveal how far each input is from each boundary, so the substitute learns much more per query than from labels alone. For language models, research has gone further: Carlini and colleagues in 2024 recovered a production model's embedding projection layer using log-probability features.
Carry it forwardOutput detail is what makes extraction cheap.
Weak answerExposing full scores by default.
Constraint changes
What if they spread queries across many accounts?
What do the accounts share?
Per-account limits are then sidestepped. Cluster accounts by payment details, network, timing and query style, look for systematic coverage of input space and synthetic-looking inputs, and add verification for high-volume tiers so creating accounts costs more.
Carry it forwardDetect extraction across accounts, not only per key.
Weak answerRelying on per-key rate limits.
Constraint changes
Which controls reduce extraction?
Which controls raise the cost per useful query?
Per-key rate and volume caps, less output detail (labels or rounded top-k scores), query-pattern monitoring, account verification, terms that prohibit training competing models with enforcement, and watermarking or fingerprinting to show a copy derives from yours. None prevents extraction outright; together they raise its cost.
Carry it forwardLayer controls to raise extraction cost.
Weak answerExpecting one control to stop extraction.
Constraint changes
What if you suspect a competitor's product is a copy?
How would you show a model was derived from yours?
Fingerprints or watermarks in your model's behaviour can support a claim that their model derives from yours. Gather evidence carefully, involve legal, and rely on terms of service and contracts. Technical evidence is probabilistic and should be presented with its confidence.
Carry it forwardFingerprinting supports claims; legal terms enforce them.
Weak answerPublicly accusing without evidence.
Now defend it
How would you protect the API?
Default to labels or rounded top-k scores; per-key and per-account volume caps; verification for high-volume tiers; monitoring and cross-account clustering of query patterns; terms prohibiting training competing models with graduated enforcement; and fingerprinting to support claims. Accept that the goal is raising cost, not preventing extraction completely.
How it's scored
Weak: "Stealing a model through the API" with no mechanism or defences.
Hire: Distinguishes functionality and parameter extraction and proposes rate limits and output restrictions.
Strong Hire: Cites results, explains the substitute-for-attacks gain, gives detection signals and realistic layered controls with honest limits.
Common mistakes
- Claiming extraction can be prevented.
- Exposing full log-probabilities by default.
- Forgetting the link between extraction and transferable attacks.