coding-guidance-benchmark
A/B test of guidance placed in front of a coding agent: does a skill, rules file, hook, MCP server configuration, or any mix of them change the security of the code the agent writes? Each arm is an opaque folder copied into the agent's workspace before it starts; the baseline has nothing installed. A real coding-agent harness (Claude Code, Codex CLI, Gemini CLI or OpenCode, run through Inspect AI's inspect-swe) builds the same seeded BaxBench tasks under every arm, and BaxBench's functional tests and exploits score the result. Reports the correct-and-secure rate per arm with intervals, the paired difference from baseline, functional and secure rates separately, failed builds, the tokens and time the guidance cost, and whether the agent touched the guidance at all. Use it when someone asks whether a piece of coding guidance works. It ships no guidance of its own. For a guardrail or content filter use guardrail-benchmark.
- License
- MIT
- Compatibility
- Requires Docker, git, and two Python environments: one with inspect_ai 0.3.289 and inspect-swe 0.3.0 plus the SDK of the model provider (runs the agent), and Python 3.12 with pipenv (runs BaxBench's scoring). The keyless smoke test needs only Python 3.11+.
Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/coding-guidance-benchmark/SKILL.md
- skills/coding-guidance-benchmark/arms/placebo/NOTES.md
- skills/coding-guidance-benchmark/arms/placebo/almanac/tides.csv
- skills/coding-guidance-benchmark/arms/placebo/arm.json
- skills/coding-guidance-benchmark/fixtures/fake_tasks.json
- skills/coding-guidance-benchmark/fixtures/mock-app/app.py
- skills/coding-guidance-benchmark/lib/arms.py
- skills/coding-guidance-benchmark/lib/baxbench.py
- skills/coding-guidance-benchmark/lib/baxbench_bridge.py
- skills/coding-guidance-benchmark/lib/evalkit.py
- skills/coding-guidance-benchmark/lib/fake_scorer.py
- skills/coding-guidance-benchmark/lib/inspect_harness.py
- skills/coding-guidance-benchmark/lib/mock_harness.py
- skills/coding-guidance-benchmark/lib/stats.py
- skills/coding-guidance-benchmark/lib/tasks.py
- skills/coding-guidance-benchmark/lib/transcript.py
- skills/coding-guidance-benchmark/new_experiment.sh
- skills/coding-guidance-benchmark/references/arm-format.md
- skills/coding-guidance-benchmark/references/baxbench-integration.md
- skills/coding-guidance-benchmark/report.py
- skills/coding-guidance-benchmark/run_benchmark.py
- skills/coding-guidance-benchmark/tests/test_arms.py
- skills/coding-guidance-benchmark/tests/test_run_benchmark.py
- skills/coding-guidance-benchmark/tests/test_scoring.py
- skills/coding-guidance-benchmark/tests/test_stats.py
- skills/coding-guidance-benchmark/tests/test_tasks.py
- skills/coding-guidance-benchmark/tests/test_touched.py
Every link opens the file at its source, pinned to the revision this page describes.