wtcooper/defense-evals
Evaluate how well a defence works: guardrails, guidance placed in front of a coding agent, and security tools, each measured against a baseline or known answers.
A/B test of guidance placed in front of a coding agent: does a skill, rules file, hook, MCP server configuration, or any mix of them change the security of the code the agent writes? Each arm is an opaque folder copied into the agent's workspace before it starts; the baseline has nothing installed. A real coding-agent harness (Claude Code, Codex CLI, Gemini CLI or OpenCode, run through Inspect AI's inspect-swe) builds the same seeded BaxBench tasks under every arm, and BaxBench's functional tests and exploits score the result. Reports the correct-and-secure rate per arm with intervals, the paired difference from baseline, functional and secure rates separately, failed builds, the tokens and time the guidance cost, and whether the agent touched the guidance at all. Use it when someone asks whether a piece of coding guidance works. It ships no guidance of its own. For a guardrail or content filter use guardrail-benchmark.
Measure how well a guardrail works, two ways. DIRECT mode sends a bundled labelled corpus (harmful vs benign, plus flattened multi-turn) straight to the guardrail or classifier API, with no model in the loop, and scores its own verdict: catch rate against false-block rate, by technique. IN-LINE mode runs standard Inspect benchmarks with the same model behind each guardrail arm and reports the risk each guardrail removes beside what it costs in over-blocking, paired on the same samples. Use to characterise one guardrail, compare vendors, or show the effect of enabling one. The guardrail is supplied by the user. For an app endpoint use app-benchmark; for comparing models use model-benchmark.
Compare security tools against targets with KNOWN answers, to decide which tool to rely on and what adding one buys over another. Three kinds, one design: code compares code scanners (CodeQL, Semgrep, Trivy, osv-scanner, zizmor, an LLM code reviewer) on source with known vulnerabilities; extension compares scanners of AI components (skills, plugins, MCP servers, model files) on labelled malicious and benign samples; pentest compares black-box pentest tools (an AI pentest agent, Nuclei, ZAP) against running vulnerable apps, counting only oracle-confirmed exploits. Tool-agnostic: each tool is driven by a small adapter the user picks. Reports recall, precision or false-positive rate with intervals, what each tool finds that the others miss, and run-to-run agreement for nondeterministic tools. It does not scan your own code or components.