security-tool-benchmark
Compare security tools against targets with KNOWN answers, to decide which tool to rely on and what adding one buys over another. Three kinds, one design: code compares code scanners (CodeQL, Semgrep, Trivy, osv-scanner, zizmor, an LLM code reviewer) on source with known vulnerabilities; extension compares scanners of AI components (skills, plugins, MCP servers, model files) on labelled malicious and benign samples; pentest compares black-box pentest tools (an AI pentest agent, Nuclei, ZAP) against running vulnerable apps, counting only oracle-confirmed exploits. Tool-agnostic: each tool is driven by a small adapter the user picks. Reports recall, precision or false-positive rate with intervals, what each tool finds that the others miss, and run-to-run agreement for nondeterministic tools. It does not scan your own code or components.
- License
- MIT
- Compatibility
- Requires python>=3.11 and git, plus the tools you choose to compare. The extension kind uses Docker with no network for model files; the pentest kind uses Docker for targets.
Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/security-tool-benchmark/SKILL.md
- skills/security-tool-benchmark/code/adapters/README.md
- skills/security-tool-benchmark/code/adapters/codeql.sh
- skills/security-tool-benchmark/code/adapters/osv-scanner.sh
- skills/security-tool-benchmark/code/adapters/replay.sh
- skills/security-tool-benchmark/code/adapters/sdlc-scan-code.sh
- skills/security-tool-benchmark/code/adapters/semgrep.sh
- skills/security-tool-benchmark/code/adapters/trivy.sh
- skills/security-tool-benchmark/code/adapters/zizmor.sh
- skills/security-tool-benchmark/code/fetch_targets.sh
- skills/security-tool-benchmark/code/match.py
- skills/security-tool-benchmark/code/materialize.sh
- skills/security-tool-benchmark/code/mutate.py
- skills/security-tool-benchmark/code/run_scanner.sh
- skills/security-tool-benchmark/code/translate_gt.py
- skills/security-tool-benchmark/ephemeral_repo.sh
- skills/security-tool-benchmark/extension/Dockerfile
- skills/security-tool-benchmark/extension/adapters/README.md
- skills/security-tool-benchmark/extension/adapters/_common.sh
- skills/security-tool-benchmark/extension/adapters/aig-mcp-scan.sh
- skills/security-tool-benchmark/extension/adapters/aig-skill-scan.sh
- skills/security-tool-benchmark/extension/adapters/cisco-mcp-scanner.sh
- skills/security-tool-benchmark/extension/adapters/cisco-skill-scanner.sh
- skills/security-tool-benchmark/extension/adapters/modelscan.sh
- skills/security-tool-benchmark/extension/adapters/scan_report_to_sarif.py
- skills/security-tool-benchmark/extension/adapters/sdlc-scan-mcp.sh
- skills/security-tool-benchmark/extension/adapters/sdlc-scan-model.sh
- skills/security-tool-benchmark/extension/adapters/sdlc-scan-skill.sh
- skills/security-tool-benchmark/extension/adapters/skillspector.sh
- skills/security-tool-benchmark/extension/adapters/yara.sh
- skills/security-tool-benchmark/extension/fetch_sets.sh
- skills/security-tool-benchmark/extension/mutate_pickle.py
- skills/security-tool-benchmark/extension/mutate_skill.py
- skills/security-tool-benchmark/extension/run_scanner.sh
- skills/security-tool-benchmark/extension/score_assets.py
- skills/security-tool-benchmark/extension/yara_filter.sh
- skills/security-tool-benchmark/lib/canary_probe.py
- skills/security-tool-benchmark/lib/cwe_map.py
- skills/security-tool-benchmark/lib/evalkit.py
- skills/security-tool-benchmark/lib/evalstats.py
- skills/security-tool-benchmark/lib/find_sdlc.sh
- skills/security-tool-benchmark/lib/manifest.py
- skills/security-tool-benchmark/lib/probelib.py
- skills/security-tool-benchmark/lib/sarif_to_findings.py
- skills/security-tool-benchmark/new_experiment.sh
- skills/security-tool-benchmark/pentest/adapters/README.md
- skills/security-tool-benchmark/pentest/adapters/nuclei.sh
- skills/security-tool-benchmark/pentest/adapters/strix.sh
- skills/security-tool-benchmark/pentest/adapters/zap.sh
- skills/security-tool-benchmark/pentest/run_dast.sh
- skills/security-tool-benchmark/pentest/targets/oracle.template.sh
- skills/security-tool-benchmark/pentest/validate.py
- skills/security-tool-benchmark/pentest/xben_mutate.sh
- skills/security-tool-benchmark/references/code.md
- skills/security-tool-benchmark/references/extension.md
- skills/security-tool-benchmark/references/pentest.md
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/README.md
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/aig-mcp-scan/clean-mockllm.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/aig-mcp-scan/flagged-mockllm.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/aig-skill-scan/clean-mockllm.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/aig-skill-scan/flagged-mockllm.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/aig-skill-scan/flagged-zh-severity-mockllm.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/cisco-mcp-scanner/behavioral-flagged-mockllm.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/cisco-mcp-scanner/behavioral-nothing-analysed.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/cisco-mcp-scanner/clean.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/cisco-mcp-scanner/combined-mockllm.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/cisco-mcp-scanner/flagged-poisoned-tools.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/cisco-mcp-scanner/llm-requested-but-not-run.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/cisco-skill-scanner/clean.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/cisco-skill-scanner/flagged-staged-fetch.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/cisco-skill-scanner/llm-analysis-failed-mockllm.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/mock-replies/aig-clean.txt
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/mock-replies/aig-flagged.txt
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/mock-replies/cisco-mcp-behavioral-flagged.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/mock-replies/cisco-mcp-llm-flagged.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/mock-replies/cisco-skill-llm-clean.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/mock-replies/unusable.txt
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/samples/benign-mcp/server.py
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/samples/benign-mcp/tools.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/samples/benign-skill/scripts/column_stats.py
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/samples/benign-skill/skill-md.txt
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/samples/poisoned-mcp/server.py
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/samples/poisoned-mcp/tools.json
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/skillspector/clean.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/skillspector/flagged-staged-fetch.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/skillspector/llm-replies-unusable-mockllm.sarif
- skills/security-tool-benchmark/tests/fixtures/scanner-adapters/skillspector/llm-unavailable.sarif
- skills/security-tool-benchmark/tests/fixtures/smoke/ground_truth.jsonl
- skills/security-tool-benchmark/tests/fixtures/smoke/tool-a.sarif
- skills/security-tool-benchmark/tests/fixtures/smoke/tool-b.sarif
- skills/security-tool-benchmark/tests/fixtures/smoke/tree/db.py
- skills/security-tool-benchmark/tests/fixtures/smoke/tree/download.py
- skills/security-tool-benchmark/tests/fixtures/smoke/tree/jobs.py
- skills/security-tool-benchmark/tests/fixtures/smoke/tree/util.py
- skills/security-tool-benchmark/tests/mock_openai_server.py
- skills/security-tool-benchmark/tests/smoke.sh
- skills/security-tool-benchmark/tests/test_code_kind.py
- skills/security-tool-benchmark/tests/test_extension_kind.py
- skills/security-tool-benchmark/tests/test_failed_runs.py
- skills/security-tool-benchmark/tests/test_scanner_adapters.py
Every link opens the file at its source, pinned to the revision this page describes.