wtcooper/model-evals
Evaluate a model through whatever API it has: security benchmarks for chat, vision, decision and voice models, and a guided review for image generation.
Compare decision models (TypeSafe Jev, OpenAI Decisions, Microsoft-Decision-1, Gemini Decision, Clef, any System One compatible server) used as a security gate, and measure how easily a decision can be flipped. Decision models take a state plus typed questions with a closed answer set (yes/no, choice, score) and return probabilities, no text. One adapter speaks the three wire formats; every arm gets the same seeded labelled cases and adversarial variants (injection, padding, option reorder, misleading option rename, paraphrase, encoding, extra JSON fields). Reports fail-open and fail-closed rates, flip rate per attack axis beside the identical-re-run noise floor, targeted attack success, consistency, calibration, refusal and error rates, latency and input tokens, with 95% intervals and a paired comparison between arms. Use when asked whether a decision model can gate tool calls, screen for prompt injection, route alerts or enforce a content policy. For chat models use model-benchmark.
Guided security review of an image-generation model (and, briefly, video) when someone asks "can we enable this?" and no standardized benchmark answers it. Covers the security concerns rather than content safety: phishing and brand-impersonation assets, identity documents, QR and hidden payloads, likenesses of real people, editing of uploaded documents, and whether provenance marks (C2PA metadata, invisible watermarks) are present and survive ordinary handling. Produces a structured record: what the vendor publishes, what a hand-built probe set showed (counts with 95% intervals), what a C2PA marker check found, and every threat that was not assessed. Writes evidence.jsonl rows other review skills consume. It sets no bar and makes no recommendation; the decision is left to people. For a model that reads images use model-benchmark; for agent features use agent-feature-review.
Measure and COMPARE the security posture of AI models themselves — open-weights or frontier — when a new one is released: a fixed suite of Inspect benchmarks, one per risk dimension (direct misuse, over-refusal, harmful-request refusal, jailbreak uplift, insecure code, prompt injection via content, injection via tools, memory poisoning, and injection via images for vision models), run on the same seeded samples for every model. Reports each rate with a confidence interval and a paired comparison between models. For a guardrail's effect use guardrail-benchmark; for an app endpoint use app-benchmark.
Measure whether a speech-to-speech (realtime voice) model is weaker when a request is SPOKEN than when it is typed. Sends the same seeded harmful and benign requests to one realtime model as text, as locally synthesised speech, and as that speech under audio perturbations (speed, pitch, volume, noise, echo, reverb), then has a separate judge model grade each reply transcript. Reports attack success and benign refusal per arm with 95% intervals, the paired speech-minus-text gap with exact McNemar, and per-perturbation deltas. Use it when a voice model or voice agent is under review (for example OpenAI gpt-realtime-2.1, which is reachable only on the Realtime API), or when a text-only benchmark result is being assumed to hold for a voice deployment. It also lists the voice threats that have no standard benchmark as review questions. For a text or vision model use model-benchmark; for an app endpoint use app-benchmark.