guardrail-benchmark
Measure how well a guardrail works, two ways. DIRECT mode sends a bundled labelled corpus (harmful vs benign, plus flattened multi-turn) straight to the guardrail or classifier API, with no model in the loop, and scores its own verdict: catch rate against false-block rate, by technique. IN-LINE mode runs standard Inspect benchmarks with the same model behind each guardrail arm and reports the risk each guardrail removes beside what it costs in over-blocking, paired on the same samples. Use to characterise one guardrail, compare vendors, or show the effect of enabling one. The guardrail is supplied by the user. For an app endpoint use app-benchmark; for comparing models use model-benchmark.
- License
- MIT
- Compatibility
- Direct mode requires python>=3.11 and node>=18 (promptfoo). In-line mode requires Python 3.11+ and a separate virtual environment with inspect_ai, inspect_evals and openai.
Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/guardrail-benchmark/SKILL.md
- skills/guardrail-benchmark/adapters/generic_guardrail.js
- skills/guardrail-benchmark/arms.example.json
- skills/guardrail-benchmark/connectors/__init__.py
- skills/guardrail-benchmark/connectors/base.py
- skills/guardrail-benchmark/connectors/litellm.py
- skills/guardrail-benchmark/connectors/openai_passthrough.py
- skills/guardrail-benchmark/corpus/benign_overrefusal.json
- skills/guardrail-benchmark/corpus/data_leakage.json
- skills/guardrail-benchmark/corpus/harmful.json
- skills/guardrail-benchmark/corpus/injection.json
- skills/guardrail-benchmark/env.example
- skills/guardrail-benchmark/injection_shim.py
- skills/guardrail-benchmark/install_dependencies.sh
- skills/guardrail-benchmark/lib/evalkit.py
- skills/guardrail-benchmark/lib/extract_transcript.js
- skills/guardrail-benchmark/lib/inspect_report.py
- skills/guardrail-benchmark/lib/inspect_runner.py
- skills/guardrail-benchmark/lib/inspect_suite.py
- skills/guardrail-benchmark/lib/status_policy.js
- skills/guardrail-benchmark/lib/status_policy.py
- skills/guardrail-benchmark/lib/summarize.py
- skills/guardrail-benchmark/new_experiment.sh
- skills/guardrail-benchmark/promptfooconfig.yaml
- skills/guardrail-benchmark/report.py
- skills/guardrail-benchmark/run_inline.py
- skills/guardrail-benchmark/tests/generic_guardrail.test.cjs
- skills/guardrail-benchmark/tests/mock_gateway.py
- skills/guardrail-benchmark/tests/mock_server.py
- skills/guardrail-benchmark/tests/test_connectors.py
- skills/guardrail-benchmark/tests/test_run_inline.py
- skills/guardrail-benchmark/tests/test_shim_integration.py
Every link opens the file at its source, pinned to the revision this page describes.