Skip to content

wtcooper/model-evals

v0.1.0MIT

Evaluate a model through whatever API it has: security benchmarks for chat, vision, decision and voice models, and a guided review for image generation.

model-benchmark

Measure and COMPARE the security posture of AI models themselves — open-weights or frontier — when a new one is released: a fixed suite of Inspect benchmarks, one per risk dimension (direct misuse, over-refusal, harmful-request refusal, jailbreak uplift, insecure code, prompt injection via content, injection via tools, memory poisoning, and injection via images for vision models), run on the same seeded samples for every model. Reports each rate with a confidence interval and a paired comparison between models. For a guardrail's effect use guardrail-benchmark; for an app endpoint use app-benchmark.

License
MIT
Compatibility
Requires Python 3.11+ and a separate virtual environment with inspect_ai, inspect_evals and openai.
Read SKILL.md at the source

Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.