Evaluate AI Security, From Protocol to Decision
Build a connected assessment for a fictional event-planning assistant. Audit its observations and review samples, compare defenses, interpret zero-event evidence and investigate changed scores before defending a recommendation.
A security score can be calculated correctly and still answer the wrong question. This course asks you to review the evidence behind a fictional conference-planning assistant, from the first evaluation request to a recommendation that another engineer can challenge.
The assistant reads event material and private attendee notes, prepares vendor briefs and may publish them through queued work. You will work with synthetic fixtures, event records and deliberately incomplete results. Each lesson adds a usable artifact: a review contract, fixture specification, observer audit, weighted estimate, matched comparison, study plan or correction note.
The exercises distinguish a verified defect from its frequency, a generated proposal from an actual effect, and an unobserved outcome from a denial. The final project combines those distinctions when a new measurement problem changes what the earlier results can support. Worked feedback explains both the conclusion and the evidence that could reverse it.
This is an assessment workshop. The concept library supplies the underlying definitions, and the application courses cover control design and operational incident response. Here you learn to choose observations, check calculations and state a defensible claim. The course does not certify a system or substitute a score for release authority.
The first two lessons are open. Sign in for the observer audit; the remaining lessons develop weighted review, comparison, study planning and the final evidence packet. All code is a small offline model or calculation using synthetic data, with its limitations stated beside it.
When you finish, you can
- Translate a product decision into requirements, authorized scope and a measurement contract
- Build a synthetic fixture that retains the permission boundary and records unresolved effects
- Audit collection and scoring against evidence with explicit reference-review limits
- Estimate outcome rates from a stratified review without ignoring selection weights
- Compare matched security outcomes alongside useful permitted work
- Plan and interpret a fixed-sample zero-event bound under stated assumptions
- Investigate score changes without confusing scorer drift, case mix and target behavior
- Defend and revise an assessment recommendation from a connected evidence packet
BEFORE YOU START · You can follow an application request and read basic Python, fractions and percentages. Read the free AI red-team introduction if assessment scope is unfamiliar. The course uses fictional records and offline calculations; no provider account, scanner or production access is required.
Syllabus
Set the decision and test contract
Turn a product question into an authorized review, then build a fixture whose observations preserve the boundary being evaluated.
Audit observations and review samples
Check whether the collector and scorer measure the intended event, then correct estimates for how review cases were selected.
Compare candidates and bound claims
Read paired results alongside useful work, and plan a zero-event study whose sample and stopping rule support its stated conclusion.
Investigate changes and defend the assessment
Diagnose a changed score, correct the evidence record and defend a bounded recommendation from the complete review packet.