Skip to content

wtcooper/defense-evals

v0.1.0MIT

Evaluate how well a defence works: guardrails, guidance placed in front of a coding agent, and security tools, each measured against a baseline or known answers.

security-tool-benchmark

Compare security tools against targets with KNOWN answers, to decide which tool to rely on and what adding one buys over another. Three kinds, one design: code compares code scanners (CodeQL, Semgrep, Trivy, osv-scanner, zizmor, an LLM code reviewer) on source with known vulnerabilities; extension compares scanners of AI components (skills, plugins, MCP servers, model files) on labelled malicious and benign samples; pentest compares black-box pentest tools (an AI pentest agent, Nuclei, ZAP) against running vulnerable apps, counting only oracle-confirmed exploits. Tool-agnostic: each tool is driven by a small adapter the user picks. Reports recall, precision or false-positive rate with intervals, what each tool finds that the others miss, and run-to-run agreement for nondeterministic tools. It does not scan your own code or components.

License
MIT
Compatibility
Requires python>=3.11 and git, plus the tools you choose to compare. The extension kind uses Docker with no network for model files; the pentest kind uses Docker for targets.
Read SKILL.md at the source

Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.