Skip to content
AI Security

Interview question · LLM Application Security

Practical coding: implement a rate limiter for an LLM API that budgets tokens, not requests, then make it work across many gateway nodes.

Requests per minute is the wrong unit when one request can cost ten thousand tokens. The interview checks the bucket arithmetic, the unknown output length, and what happens when two gateway nodes share one budget.

mediumNewPreparation tags: OpenAI, Anthropic, AmazonUpdated Nov 2026

Requests per minute is the wrong unit when one request can cost ten thousand tokens. The interview checks the bucket arithmetic, the unknown output length, and what happens when two gateway nodes share one budget.

Make your first attempt

What would you say before reading the answer?

  1. 01 State the assumptions your answer depends on.
  2. 02 Explain the mechanism and choose a control.
  3. 03 Name a condition that would change your decision.

The worked answer includes an approach, scoring criteria and follow-up questions. Compare the reasoning with your own.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 529 remaining answers · $25

Every worked answer in all 17 topics, every lesson in every course, the full concept curriculum, the complete practice-test bank, and the downloadable security field guides.

6 months · One payment · No auto-renewal

What if · the follow-ups in this answer

The answer works through 6 follow-ups like an interviewer would

An LLM API needs rate limiting that allows short bursts but caps sustained use per customer, across many gateway nodes.

  1. How does a token bucket work?
  2. What should a token represent for an LLM API?
  3. Where should the limiter run?
  4. What if the shared store becomes unavailable?
  5. How do you communicate limits to customers?
  6. What would the implementation include?

Each rung has a nudge, a model answer, the principle that carries over and the weak answer to avoid. Premium opens every answer and ladder.

UP NEXT ON YOUR JOURNEY
Next in this trackYour CISO wants to buy an AI security platform. How do you evaluate one, module by module, and what will it never do for you?Next in this trackHow do AI providers detect and disrupt attackers using their models, and how would you build that capability for your own AI platform?Popular right nowWhat security risks does the Model Context Protocol (MCP) introduce?
DISCUSSION · 0

No comments yet. Share a correction or a different way to answer it.