model-benchmark
Measure and COMPARE the security posture of AI models themselves — open-weights or frontier — when a new one is released: a fixed suite of Inspect benchmarks, one per risk dimension (direct misuse, over-refusal, harmful-request refusal, jailbreak uplift, insecure code, prompt injection via content, injection via tools, memory poisoning, and injection via images for vision models), run on the same seeded samples for every model. Reports each rate with a confidence interval and a paired comparison between models. For a guardrail's effect use guardrail-benchmark; for an app endpoint use app-benchmark.
- License
- MIT
- Compatibility
- Requires Python 3.11+ and a separate virtual environment with inspect_ai, inspect_evals and openai.
Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/model-benchmark/SKILL.md
- skills/model-benchmark/lib/evalkit.py
- skills/model-benchmark/lib/inspect_report.py
- skills/model-benchmark/lib/inspect_runner.py
- skills/model-benchmark/lib/inspect_suite.py
- skills/model-benchmark/new_experiment.sh
- skills/model-benchmark/report.py
- skills/model-benchmark/run_suite.py
- skills/model-benchmark/tests/test_model_benchmark.py
Every link opens the file at its source, pinned to the revision this page describes.