What is attack success rate, and how do you report findings when results are probabilistic?

TL;DR: Define the event and trial unit, then report successes, resolved non-successes, unresolved outcomes and execution gaps under a versioned protocol. Add an uncertainty estimate only when its assumptions match the sampling process. Separate observed frequency from finding severity and evaluate adaptive retry budgets as their own process rather than automatically compounding a pooled percentage.

How to approach it

Start with a result that can be observed, such as a protected field reaching a forbidden recipient. Name the actor, configuration and budget. Explain the numerator and denominator before introducing an interval, and distinguish the run's descriptive result from an estimate for future cases.

A strong answer

In an original fictional assessment, a support assistant handles synthetic customer orders. The tester controls one customer's messages and attachments. Another customer's private note must not reach that customer's reply or readable draft. The scope is authorized and isolated; controlled receivers observe admitted effects after the application checks.

The metric is the fraction of started, resolved conversations that produce a verified prohibited release. A proposed tool call is not the release. If retrieval itself violates a separate rule, report that finding too, while keeping its count distinct from recipient disclosure.

Reconcile the counts. Suppose twenty-four selected conversations all start and resolve, each with up to five user turns. Four conversations cause the defined release. They contain fifty-four user messages in total. The observed conversation success rate is 4/24, about 16.7%. Dividing four successful conversations by fifty-four messages mixes units and does not answer the conversation-level question.

The turn budget, stopping rule and actual resource use belong in the report. If a conversation stops early after the objective occurs, that is a protocol outcome to preserve, not a reason to add imaginary unused attempts to the denominator. If an effect is unresolved, retain that state and explain how it limits the result.

Identify the sampling model. These selected conversations describe the reviewed set. They are not automatically representative of all future customers or attack campaigns. For independent binary trials with a common success probability under an appropriate sampling design, a binomial interval can describe uncertainty in that probability. The NIST handbook gives Wilson and exact interval methods.

For a separate illustrative binomial sample, two successes in twenty trials gives an approximate two-sided 95% Wilson interval of 2.8% to 30.1%. Zero in fifty gives an upper endpoint about 7.1% with that same method. A one-sided 95% exact bound is different. Name the method and sidedness rather than mixing endpoints to obtain a favorable result.

Case selection, dependence and scoring can dominate the uncertainty. Repeating many variants from one sampled document does not create the same evidence as sampling many independent documents. A fixed suite's generation variability and generalization to new scenarios are different targets. Use the statistics concept to choose an analysis that preserves that structure.

Treat retries as a defined opportunity. Under a known constant probability p and independent attempts, at least one success in k tries has probability 1 − (1 − p)^k. A measured rate from selected cases is not automatically that p. Adaptive feedback, shared state, heterogeneous tasks, rate limits and containment change the process. Measure a budgeted campaign directly when those conditions are part of the question.

Validate what counted as success. Programmatic checks need correct placement and evidence: a generated destination, a denied operation and an observed receiver event differ. Model judges need a calibrated rubric and reviewed examples. Human reference labels also need context and adjudication. Report missing effects and scorer uncertainty instead of claiming every logged event is unambiguous.

Keep the finding actionable. One verified prohibited release may require containment even if its repeat rate is unknown. Assess the supported asset, reader, access conditions and consequences separately from frequency. Also report allowed-task completion: refusing every reply can restrict exposure while preventing the intended service.

What interviewers probe next

Write a one-paragraph result for the twenty-four conversations. Include four successes, the conversation unit and five-turn cap, selected-case provenance, configuration, actual recipient evidence and the absence of a supported population or adaptive-campaign estimate. State the finding and the permitted-work retest needed after a fix.

If the new version is compared with the old, preserve paired case outcomes and consistent scoring. Separate a fixed-suite regression comparison from a fresh adaptive exercise; neither number replaces the other.

What if?

How it's scored

• Weak: Gives a percentage without its unit, counts unknown effects as denials or predicts attacker success by compounding an unexplained rate.
• Hire: Defines the event, protocol and denominator, reports uncertainty under stated assumptions and separates severity from frequency.
• Strong Hire: Reconciles execution and evidence gaps, distinguishes fixed-suite and population claims, validates the scorer and preserves paired and budgeted-campaign evidence.

Common mistakes

Dividing conversation successes by messages. Reporting a tool proposal as a completed release. Using a two-sided Wilson endpoint as an exact one-sided bound. Treating all repeated cases as independent population draws. Declaring a serious defect acceptable because its aggregate rate is small.