TL;DR: Walk the request path and threat-model each layer. Gateway: stolen or shared keys and cost exhaustion; per-key and per-tenant token, rate and spend limits. Input processing: parsers for text, images, audio and documents, plus tokenizers, all attack surface; patch, sandbox and size-limit. Model server: framework vulnerabilities, unsafe model loading, exposed debug and management endpoints. Shared state: prefix caches, batching and GPU memory across tenants; scope per tenant. Logs: sensitive prompts retained broadly. Outputs and API as oracle: extraction, abuse at scale. Weights: protected on serving nodes. Assign deterministic controls to each and abuse detection across all.
How to approach it
Clarify scope: is this a public multi-tenant API, an internal platform, or a single application's backend? Does it serve customer-uploaded models? Then draw the path, take each component through STRIDE, and prioritise by impact and likelihood. Interviewers expect cost and tenancy to feature prominently.
A strong answer
Gateway
keys, cost limits
Input processing
parsers, tokenizers
Scheduler
batching, isolation
GPU serving
memory, side channels
Outputs and logs
leakage, retention
Main threats
Stolen keys, cost exhaustion
- Per-key token, rate and spend limits
Parser bugs
- Patch, sandbox, size-limit
Cross-tenant leakage
- Batch and cache isolation
Assets. Customer prompts and outputs (confidential), model weights, service availability and spend, tenant isolation, provider reputation and abuse obligations.
Adversaries. Abusive customers, attackers with stolen keys, competitors attempting extraction, tenants probing other tenants, and remote attackers targeting exposed infrastructure.
Threats and controls by layer.
| Layer | Key threats | Controls |
|---|---|---|
| Gateway and auth | Stolen keys; key sharing; cost exhaustion; enumeration | Scoped, rotatable keys; per-key and per-tenant token, request and spend limits; anomaly detection; key leak scanning |
| Request validation | Huge prompts, max-length generations, many parallel streams | Hard caps on input and output tokens, timeouts, concurrency per tenant |
| Input parsers | Malformed images, audio, PDFs; decompression bombs; tokenizer edge cases | Patched, sandboxed parsers with size and type limits; fuzzing |
| Model server | Framework CVEs; unsafe model loading; debug endpoints | Patch SLAs; minimal images; safetensors-only loading; no debug or management endpoints exposed |
| Shared state | Cross-tenant leakage through prefix caches (timing), batch interference, GPU memory residue | Per-tenant cache scoping; dedicated capacity for sensitive tenants; memory clearing |
| Management plane | Exposed dashboards, job APIs, metrics with sensitive labels | Private networks only, authentication, network policy |
| Logs and telemetry | Prompts with personal data, long retention, broad access | Redaction, short retention, access control, customer opt-outs |
| Outputs / oracle | Model extraction; large-scale abuse generation | Rate limits, usage-pattern detection, policy enforcement |
| Weights on nodes | Theft from serving hosts | No interactive access, egress limits, encrypted storage with controlled keys |
Prioritisation. Cost exhaustion and key abuse are high likelihood and immediate impact; exposed management planes are low effort for attackers and catastrophic; cross-tenant cache leakage is lower likelihood but high impact for confidential workloads; parser bugs depend on input modalities.
Customer-uploaded models. If tenants can bring their own models or adapters, loading becomes untrusted code execution risk: isolate per tenant, safe formats only, no remote code.
Detection and response. Per-tenant usage baselines, alerts on spend spikes and unusual patterns, ability to revoke keys and throttle tenants instantly.
Open API
Leaked key, no quotas
unlimited tokens
Cost explosion, extraction
batching leaks
Bill and data exposure
Outcome, unsafe: Model copied
Tenants mixed
Controlled API
Per-key, per-tenant budgets
abuse detection
Isolated batching and caches
sandboxed parsers
Abuse capped
Outcome, safe: Tenants isolated
Extraction detected
What interviewers probe next
- "A tenant wants guaranteed isolation from others. What do you offer?" Dedicated capacity with no shared caches or GPUs, separate logging, and contractual retention terms.
- "How do you detect extraction?" High-volume, systematic, diverse queries from a key, especially with logit or probability outputs; rate limits and restricting such outputs help.
- "What's your first week of work?" Inventory internet-exposed endpoints, enforce limits at the gateway, confirm management planes are private, and set a patch process for serving frameworks.
What if?
What if · interview follow-ups
Hold your answer as the constraints change
A public LLM inference API accepts text, images and documents from many customers at scale.
Start here
What threats sit at the gateway?
What would an attacker with a stolen key do?
Stolen or shared keys and cost exhaustion. Apply per-key and per-tenant token, rate and spend limits, authenticate early, and reject oversized requests before they reach expensive services.
Carry it forwardGateways enforce identity, limits and budgets.
Weak answerCounting requests instead of tokens and compute.
Constraint changes
What about input processing?
What code touches the uploaded file first?
Parsers for images, audio and documents, plus tokenizers, are attack surface. Patch them, run them sandboxed with resource limits, and size-limit inputs so a malformed file cannot crash or compromise the service.
Carry it forwardParsers are attack surface; sandbox them.
Weak answerParsing documents in the serving process.
Constraint changes
What about the model server?
Which endpoints should never face the internet?
Framework vulnerabilities, unsafe model loading and exposed debug or management endpoints. Keep management interfaces private, load only approved artefacts in safe formats, and patch serving frameworks.
Carry it forwardKeep management planes private and loading safe.
Weak answerExposing the serving framework's admin port.
Constraint changes
What about shared state between tenants?
What persists between requests from different customers?
Prefix caches, batching and GPU memory can carry one tenant's data near another's. Scope caches per tenant, clear memory between uses where isolation is weak, and test for cross-tenant leakage.
Carry it forwardScope shared state per tenant.
Weak answerGlobal prompt caches across tenants.
Constraint changes
What about logs and outputs?
Who can read the prompt logs?
Logs may retain sensitive prompts broadly, so restrict and minimise them. Outputs make the API an oracle for extraction and abuse at scale, so monitor usage patterns, enforce terms and limit output detail such as log probabilities.
Carry it forwardProtect logs and watch the API as an oracle.
Weak answerLogging every prompt to a shared platform.
Now defend it
How would you summarise the threat model?
Walk the request path: gateway (key theft, cost exhaustion; per-key and tenant limits), input processing (parser bugs; sandboxing and size limits), model server (framework bugs, unsafe loading, exposed admin; private management, safe formats, patching), shared state (caches and GPU memory; tenant scoping), logs (restricted, minimised) and outputs (extraction and abuse; monitoring and output limits).
How it's scored
Weak: Focuses on prompt injection in the abstract.
Hire: Walks the layers with concrete controls, including quotas and parser risks.
Strong Hire: Prioritises by likelihood and impact, covers multi-tenancy and shared caches carefully, the management plane, customer-uploaded models, and detection and response.
Common mistakes
- Ignoring cost as an attack vector.
- Sharing prefix caches across tenants without considering leakage.
- Exposing dashboards or job APIs "temporarily".