wtcooper/defense-evals
v0.1.0MIT
Evaluate how well a defence works: guardrails, guidance placed in front of a coding agent, and security tools, each measured against a baseline or known answers.
scripts
- coding-guidance-benchmark
- fixtures/mock-app/app.py
- lib/arms.py
- lib/baxbench.py
- lib/baxbench_bridge.py
- lib/evalkit.py
- lib/fake_scorer.py
- lib/inspect_harness.py
- lib/mock_harness.py
- lib/stats.py
98 more
- lib/tasks.py
- lib/transcript.py
- new_experiment.sh
- report.py
- run_benchmark.py
- tests/test_arms.py
- tests/test_run_benchmark.py
- tests/test_scoring.py
- tests/test_stats.py
- tests/test_tasks.py
- tests/test_touched.py
- guardrail-benchmark
- adapters/generic_guardrail.js
- connectors/__init__.py
- connectors/base.py
- connectors/litellm.py
- connectors/openai_passthrough.py
- injection_shim.py
- install_dependencies.sh
- lib/evalkit.py
- lib/extract_transcript.js
- lib/inspect_report.py
- lib/inspect_runner.py
- lib/inspect_suite.py
- lib/status_policy.js
- lib/status_policy.py
- lib/summarize.py
- new_experiment.sh
- report.py
- run_inline.py
- tests/generic_guardrail.test.cjs
- tests/mock_gateway.py
- tests/mock_server.py
- tests/test_connectors.py
- tests/test_run_inline.py
- tests/test_shim_integration.py
- security-tool-benchmark
- code/adapters/codeql.sh
- code/adapters/osv-scanner.sh
- code/adapters/replay.sh
- code/adapters/sdlc-scan-code.sh
- code/adapters/semgrep.sh
- code/adapters/trivy.sh
- code/adapters/zizmor.sh
- code/fetch_targets.sh
- code/match.py
- code/materialize.sh
- code/mutate.py
- code/run_scanner.sh
- code/translate_gt.py
- ephemeral_repo.sh
- extension/adapters/_common.sh
- extension/adapters/aig-mcp-scan.sh
- extension/adapters/aig-skill-scan.sh
- extension/adapters/cisco-mcp-scanner.sh
- extension/adapters/cisco-skill-scanner.sh
- extension/adapters/modelscan.sh
- extension/adapters/scan_report_to_sarif.py
- extension/adapters/sdlc-scan-mcp.sh
- extension/adapters/sdlc-scan-model.sh
- extension/adapters/sdlc-scan-skill.sh
- extension/adapters/skillspector.sh
- extension/adapters/yara.sh
- extension/fetch_sets.sh
- extension/mutate_pickle.py
- extension/mutate_skill.py
- extension/run_scanner.sh
- extension/score_assets.py
- extension/yara_filter.sh
- lib/canary_probe.py
- lib/cwe_map.py
- lib/evalkit.py
- lib/evalstats.py
- lib/find_sdlc.sh
- lib/manifest.py
- lib/probelib.py
- lib/sarif_to_findings.py
- new_experiment.sh
- pentest/adapters/nuclei.sh
- pentest/adapters/strix.sh
- pentest/adapters/zap.sh
- pentest/run_dast.sh
- pentest/targets/oracle.template.sh
- pentest/validate.py
- pentest/xben_mutate.sh
- tests/fixtures/scanner-adapters/samples/benign-mcp/server.py
- tests/fixtures/scanner-adapters/samples/benign-skill/scripts/column_stats.py
- tests/fixtures/scanner-adapters/samples/poisoned-mcp/server.py
- tests/fixtures/smoke/tree/db.py
- tests/fixtures/smoke/tree/download.py
- tests/fixtures/smoke/tree/jobs.py
- tests/fixtures/smoke/tree/util.py
- tests/mock_openai_server.py
- tests/smoke.sh
- tests/test_code_kind.py
- tests/test_extension_kind.py
- tests/test_failed_runs.py
- tests/test_scanner_adapters.py
plugin.json
- name
- defense-evals
- version
- 0.1.0
- description
- Evaluate how well a defence works: guardrails, guidance placed in front of a coding agent, and security tools, each measured against a baseline or known answers.
- author
- ai-security-evals contributors
- license
- MIT
plugin.json
{
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
"name": "defense-evals",
"version": "0.1.0",
"description": "Evaluate how well a defence works: guardrails, guidance placed in front of a coding agent, and security tools, each measured against a baseline or known answers.",
"author": {
"name": "ai-security-evals contributors"
},
"homepage": "https://github.com/wtcooper/ai-security-evals",
"repository": "https://github.com/wtcooper/ai-security-evals",
"license": "MIT",
"keywords": [
"ai-security",
"evaluation",
"guardrail",
"scanner",
"a-b-test"
]
}
What else this package ships
These files come with the package and this site does not publish them. They are listed so you know what is there before you install it.
- 64Python files
- 37shell scripts
- 4JavaScript files
- 18JSON files
- 4text files
- tides.csv
- promptfooconfig.yaml
- 17other files