Keep the real boundary inside the test fixture

TL;DR: Use synthetic records and controlled receivers while retaining the identity, permission and execution checks the assessment is meant to test. Bind observations to the correct run and operation, and keep pending or missing effects unresolved instead of scoring them as safe.

Where you are. Module one, lesson two. You will turn the review charter into a fixture contract and a small observation ledger.

Describe the system the fixture represents

Continue with the fictional conference assistant. A staff planner may read private attendee notes for an internal task. A vendor brief may use approved public event information, but a vendor must not gain access to private notes through the brief, preview or saved history.

The fixture uses synthetic accounts, records and recipients in the approved isolated environment. It retains the application’s relevant identity and policy checks, context assembly, queue and worker behavior. The external publication receiver is controlled by the assessment team after those checks, so an observed release has a known destination without involving a real vendor.

A permissive stub before authorization would answer a different question. Document each substitution and the claim it limits. If the real permission service is unavailable, the team may still run a component test, but must not label its result as evidence about the omitted service.

Give each case an observation contract

A case needs more than input text. Name the requirement, actor, source permissions, recipient, starting state, expected observations and relevant execution budget. Record whether the case was selected for discovery, included in a fixed comparison or sampled under a declared procedure.

For this exercise, use one stable operation ID for each intended publication. Attempts and retries may have their own IDs, but they remain linked to that operation. A run ID distinguishes otherwise similar executions. Neither identifier grants permission; it only supports the evidence join.

Useful permitted work needs its own rubric. The brief must contain the approved event information and reach the intended authorized reader. Disabling publication may prevent one effect while failing the product task. Report both observations rather than hiding one inside an overall pass label.

Reconcile the right operation

The following pure offline example joins synthetic terminal observations. It does not authenticate logs, prove their completeness or implement a production collector. Assume the fixture supplies reviewed records and that a terminal denial applies to the named publication operation. Other read and preview boundaries still need their own checks.

def summarize_effect(run_id, operation_id, records):
    states = set()
    for record in records:
        if (record.get("run"), record.get("operation")) != (
            run_id, operation_id
        ):
            continue
        state = record.get("state")
        if state not in {"accepted", "denied", "released"}:
            raise ValueError("invalid receipt state")
        states.add(state)
    if "released" in states:
        if "denied" in states:
            return "release observed; conflicting denial"
        return "release observed"
    if "denied" in states:
        return "denial observed"
    return "unresolved"


records = [
    {"run": "r1", "operation": "p7", "state": "denied"},
    {"run": "r2", "operation": "p7", "state": "released"},
    {"run": "r3", "operation": "p7", "state": "accepted"},
    {"run": "r4", "operation": "p7", "state": "denied"},
    {"run": "r4", "operation": "p7", "state": "released"},
]
print(summarize_effect("r1", "p7", records))  # => denial observed
print(summarize_effect("r2", "p7", records))  # => release observed
print(summarize_effect("r3", "p7", records))  # => unresolved
print(summarize_effect("r1", "missing", records))  # => unresolved
print(summarize_effect("r4", "p7", records))  # => release observed; conflicting denial
bad = [{"run": "r5", "operation": "p7", "state": "probably fine"}]
try:
    summarize_effect("r5", "p7", bad)
except ValueError:
    print("invalid receipt state")  # => invalid receipt state

The same operation label appears in different runs. Joining on that label alone would attach the second run’s release to the first run’s denial. The third run remains unresolved despite having an acceptance record. The conflicting fourth run retains the observed release while flagging inconsistent evidence; it must not become a safe result merely because a denial also appears.

A release is not automatically an unauthorized release. Compare its actual content and reader with the requirement. In a deployed collector, source authentication, record integrity, coverage, event ordering and downstream semantics need separate review. This small exercise isolates the join and state distinction so you can inspect it directly.

Preserve the state that makes the case meaningful

For a permission-change case, begin with the authorized state, queue the synthetic operation under the approved procedure, change the relevant permission and observe the worker’s decision. Do not clear the queue or warm cache simply to make every test start look identical. The retained state is part of this case’s contract.

For a study that requires independent fresh starts, reset the declared state and verify that reset. These are different protocols, not competing universal rules. Record environmental changes and unavailable provider version information rather than claiming a reproducible configuration you cannot identify.

Validate observation coverage using known outcomes before evaluating a candidate. If the receiver stops recording, pause affected measurement and retain the gap. A quiet collector is not evidence that all attempted releases were denied.

Do this before moving on

Create a fixture card for each of the five conditions in the table. Include the requirement, starting state, actual check retained, controlled observation point and the result that would remain unknown.

Then inspect the offline records without running the code. Predict all six outputs and explain why the fourth run cannot be called safe. Add a hypothetical preview disclosure to the first run: its final publication denial remains true, but the broader vendor-exposure requirement now fails.

Your completed artifact should let another reviewer identify what the fixture measures and what it leaves outside scope. Reject any conclusion that relies only on the model’s description of its own action.

Go deeper

• Agentic AI security traces the application permissions and execution path that the fixture must preserve.
• Test injection safely explains why controlled effects must retain the relevant enforcement path.
• AI incident response develops the distinction between verified effects, hypotheses and unknown work.
• Telemetry for AI workflows helps connect trusted identity, context, permission and destination evidence.
• Regression tests in CI turns a requirement and fixture into a maintainable change check.
• Stopping an AI feature examines what can remain in flight after new work is held.

Key takeaways

• A safe receiver can replace an external destination without replacing the tested permission check.
• Join evidence to the correct run and intended operation.
• A denial at one point does not erase an earlier unauthorized read or preview.
• Retain unresolved effects and the state relevant to the case.