Skip to content

wtcooper/defense-evals

v0.1.0MIT

Evaluate how well a defence works: guardrails, guidance placed in front of a coding agent, and security tools, each measured against a baseline or known answers.

coding-guidance-benchmark

A/B test of guidance placed in front of a coding agent: does a skill, rules file, hook, MCP server configuration, or any mix of them change the security of the code the agent writes? Each arm is an opaque folder copied into the agent's workspace before it starts; the baseline has nothing installed. A real coding-agent harness (Claude Code, Codex CLI, Gemini CLI or OpenCode, run through Inspect AI's inspect-swe) builds the same seeded BaxBench tasks under every arm, and BaxBench's functional tests and exploits score the result. Reports the correct-and-secure rate per arm with intervals, the paired difference from baseline, functional and secure rates separately, failed builds, the tokens and time the guidance cost, and whether the agent touched the guidance at all. Use it when someone asks whether a piece of coding guidance works. It ships no guidance of its own. For a guardrail or content filter use guardrail-benchmark.

License
MIT
Compatibility
Requires Docker, git, and two Python environments: one with inspect_ai 0.3.289 and inspect-swe 0.3.0 plus the SDK of the model provider (runs the agent), and Python 3.12 with pipenv (runs BaxBench's scoring). The keyless smoke test needs only Python 3.11+.
Read SKILL.md at the source

Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.