wtcooper/ai-review
Guided security review of an AI capability: scope it, gather vendor evidence, model the threats, set your own acceptance criteria, and compile the report. Measures and records; never decides.
Fifth stage of a guided AI security review. Walks an organisation through writing its own acceptance criteria into criteria.json. For each measured dimension it explains what the number means, how wide its interval is at the depth run, and what each kind of error costs in this deployment, and offers the organisation's own earlier results as reference points. Records who set each criterion, when, and whether results for the subject had been seen. Use before results are read where possible, and before review-report. Never proposes a value: the shipped template is empty and the tool has no defaults.
Second stage of a guided AI security review. Harvests what already exists about the subject before anything is measured: vendor system and model cards, published benchmark scores, documented limitations, data retention and hosting, known incidents and research, and for open weights the scan verdicts Hugging Face publishes. Each item becomes a row in evidence.jsonl with its source, capture date and provenance; what was looked for and not found is recorded as a row too. Use after review-scope and before measuring. Records evidence; does not weigh it or reach a verdict.
Final stage of a guided AI security review. Compiles report.md from review.json, evidence.jsonl, the experiment folders the measurement skills wrote under .evals/, the test plan, and criteria.json if the organisation wrote one. Sections: what was reviewed and in what context; evidence with provenance; results with each skill's validity flags surfaced first; threats not covered; the organisation's criteria beside the results (only when criteria.json exists); and decision, conditions and sign-off left blank for people. Use when the measurements a review planned are done. Compiles; never decides.
First stage of a guided AI security review. Interviews the requester about the AI capability being enabled (a model, a model inside an agent harness, an app, a decision, voice or image model, or a new agent feature) and writes review.json: the subject, the deployment context, who controls each input channel, what the capability can do, what happens on failure and the time budget. Ships a validator that prints what is missing. Use when a review is requested and before any threat model, evidence harvest or measurement. Records the scope; never judges it.
Third stage of a guided AI security review. Produces the threat model for the subject type in review.json (model, agent in a harness, app, decision model, voice model, image model, agent feature) with OWASP LLM Top 10 2025, OWASP Agentic Top 10 and MITRE ATLAS ids where verified, and a test plan that sorts every threat into: covered by a catalogued benchmark (skill, benchmark, depth), needs a custom probe, design review only, or uncovered. Use after review-scope and before running any measurement skill. Names what to measure; prescribes no controls and sets no acceptance criteria.