Explain tokens, context windows and sampling temperature, and why non-determinism changes how you write security tests.

TL;DR: Tokens represent model input and output; context limits constrain a request under its interface’s accounting rules; temperature affects a supported sampling procedure. Those settings do not define the security test. Define the requirement, case selection, configuration and observed effect, then use repetitions and uncertainty methods that fit the actual experiment.

How to approach it

Give each term a short definition and identify which changes could alter the result. Separate repeated execution of one case from coverage of new cases. Explain how you would test a required application control and how you would interpret model behavior without treating either a refusal or an average as release authority.

A strong answer

Text tokenizers map text into model-specific units, often including word pieces and punctuation. Character counts and word counts are not interchangeable with token counts. Multimodal inputs have their own supported representations and accounting. The context limit constrains the material available for the request; stored history, retrieved facts and trained parameters are separate mechanisms.

Temperature changes the relative distribution used in supported sampling. Lower values generally concentrate selection; higher values spread it, subject to the decoder and other settings. Greedy decoding and sampling are distinct choices: the Hugging Face generation documentation, checked November 2026, documents them separately. Do not assume every interface accepts the same parameter or interprets zero in the same way.

Reducing decoding randomness does not freeze the application. Retrieval results, context selection, model or service revisions, queues, permissions and concurrent operations may change. A seed is useful configuration evidence, not a guarantee that the whole system recreates an earlier execution. Record what the integration actually identifies and what remains outside its control.

Use an original synthetic result to make the test design concrete. Six selected scenarios run ten times each with complete effect observation. One scenario has two verified prohibited disclosures and the other five have none. The descriptive run fraction is 2/60, about 3.3%. One of six selected scenarios has at least one observed failure, about 16.7%. These answer different questions.

Neither automatically estimates customer incidence. The cases were selected, repeats may have different conditional failure probabilities, and shared state may create dependence. A confidence interval needs a defined target population and an analysis that respects the design. More runs on the same six cases can improve information about those conditions while leaving other scenarios unexplored. See statistics for AI evaluations.

Test application requirements directly. An authorized fixture can submit an invalid resource request to the actual handler, change a permission while work waits, or attempt a disallowed recipient. These checks should establish the required rejection in the tested conditions independently of whether the model happens to propose the operation. Code-based enforcement still needs tests for races, stale data and incomplete observation; “deterministic code” is not proof of correctness.

Also evaluate the full model-driven workflow using permitted synthetic data. Record the actual input selection, proposal, permission decision and downstream effect needed for the requirement. A refused final answer can follow a completed disclosure. A timed-out operation can remain unresolved. Keep those states distinct instead of assigning every run a convenient pass label.

A confirmed non-waivable boundary violation requires action even when its frequency is low or uncertain. Averages can describe evidence and compare compatible conditions; they cannot grant an exception the requirement does not allow. Measure useful permitted work alongside failed requirements, refusals and incomplete tasks.

Choose repetitions from the decision. A regression fixture for a known defect, a matched comparison and a sampled population estimate need different designs. Specify observation and stopping rules before drawing the intended statistical conclusion. Avoid “always run 200 times” or adding repeats until the latest result looks favorable. The evaluation course works through those choices with connected evidence.

What interviewers probe next

• “Does temperature zero make the system safe?” No. Reduced sampling variation neither authorizes actions nor freezes context and downstream state; verify the actual configuration and requirement.
• “How many runs are enough?” Define the decision, population or fixed set, dependence, effect frequency assumptions and required precision or detection goal first.
• “The same text was produced both times. Why did the outcome differ?” Inspect permissions, tool state, concurrency and observation; identical generated content can enter different application conditions.

What if?

How it's scored

Weak: Defines terms and claims one refusal or a fixed seed proves security.

Hire: Records configuration, separates cases from repetitions and checks the actual application effect.

Strong Hire: Selects a justified analysis unit, preserves missing observations, tests controls independently and refuses to average away mandatory defects or broken useful work.

Common mistakes

• Equating characters, words and tokens.
• Treating repeated runs as independent new scenarios without justification.
• Using decoding settings to explain every downstream difference.
• Declaring a known defect repaired because a later sample contains no observed failure.