Skip to content

wtcooper/defense-evals

v0.1.0MIT

Evaluate how well a defence works: guardrails, guidance placed in front of a coding agent, and security tools, each measured against a baseline or known answers.

scripts

  • coding-guidance-benchmark
  • fixtures/mock-app/app.py
  • lib/arms.py
  • lib/baxbench.py
  • lib/baxbench_bridge.py
  • lib/evalkit.py
  • lib/fake_scorer.py
  • lib/inspect_harness.py
  • lib/mock_harness.py
  • lib/stats.py
  • 98 more
    • lib/tasks.py
    • lib/transcript.py
    • new_experiment.sh
    • report.py
    • run_benchmark.py
    • tests/test_arms.py
    • tests/test_run_benchmark.py
    • tests/test_scoring.py
    • tests/test_stats.py
    • tests/test_tasks.py
    • tests/test_touched.py
    • guardrail-benchmark
    • adapters/generic_guardrail.js
    • connectors/__init__.py
    • connectors/base.py
    • connectors/litellm.py
    • connectors/openai_passthrough.py
    • injection_shim.py
    • install_dependencies.sh
    • lib/evalkit.py
    • lib/extract_transcript.js
    • lib/inspect_report.py
    • lib/inspect_runner.py
    • lib/inspect_suite.py
    • lib/status_policy.js
    • lib/status_policy.py
    • lib/summarize.py
    • new_experiment.sh
    • report.py
    • run_inline.py
    • tests/generic_guardrail.test.cjs
    • tests/mock_gateway.py
    • tests/mock_server.py
    • tests/test_connectors.py
    • tests/test_run_inline.py
    • tests/test_shim_integration.py
    • security-tool-benchmark
    • code/adapters/codeql.sh
    • code/adapters/osv-scanner.sh
    • code/adapters/replay.sh
    • code/adapters/sdlc-scan-code.sh
    • code/adapters/semgrep.sh
    • code/adapters/trivy.sh
    • code/adapters/zizmor.sh
    • code/fetch_targets.sh
    • code/match.py
    • code/materialize.sh
    • code/mutate.py
    • code/run_scanner.sh
    • code/translate_gt.py
    • ephemeral_repo.sh
    • extension/adapters/_common.sh
    • extension/adapters/aig-mcp-scan.sh
    • extension/adapters/aig-skill-scan.sh
    • extension/adapters/cisco-mcp-scanner.sh
    • extension/adapters/cisco-skill-scanner.sh
    • extension/adapters/modelscan.sh
    • extension/adapters/scan_report_to_sarif.py
    • extension/adapters/sdlc-scan-mcp.sh
    • extension/adapters/sdlc-scan-model.sh
    • extension/adapters/sdlc-scan-skill.sh
    • extension/adapters/skillspector.sh
    • extension/adapters/yara.sh
    • extension/fetch_sets.sh
    • extension/mutate_pickle.py
    • extension/mutate_skill.py
    • extension/run_scanner.sh
    • extension/score_assets.py
    • extension/yara_filter.sh
    • lib/canary_probe.py
    • lib/cwe_map.py
    • lib/evalkit.py
    • lib/evalstats.py
    • lib/find_sdlc.sh
    • lib/manifest.py
    • lib/probelib.py
    • lib/sarif_to_findings.py
    • new_experiment.sh
    • pentest/adapters/nuclei.sh
    • pentest/adapters/strix.sh
    • pentest/adapters/zap.sh
    • pentest/run_dast.sh
    • pentest/targets/oracle.template.sh
    • pentest/validate.py
    • pentest/xben_mutate.sh
    • tests/fixtures/scanner-adapters/samples/benign-mcp/server.py
    • tests/fixtures/scanner-adapters/samples/benign-skill/scripts/column_stats.py
    • tests/fixtures/scanner-adapters/samples/poisoned-mcp/server.py
    • tests/fixtures/smoke/tree/db.py
    • tests/fixtures/smoke/tree/download.py
    • tests/fixtures/smoke/tree/jobs.py
    • tests/fixtures/smoke/tree/util.py
    • tests/mock_openai_server.py
    • tests/smoke.sh
    • tests/test_code_kind.py
    • tests/test_extension_kind.py
    • tests/test_failed_runs.py
    • tests/test_scanner_adapters.py
plugin.json
name
defense-evals
version
0.1.0
description
Evaluate how well a defence works: guardrails, guidance placed in front of a coding agent, and security tools, each measured against a baseline or known answers.
author
ai-security-evals contributors
license
MIT

What else this package ships

These files come with the package and this site does not publish them. They are listed so you know what is there before you install it.

  • 64Python files
  • 37shell scripts
  • 4JavaScript files
  • 18JSON files
  • 4text files
  • tides.csv
  • promptfooconfig.yaml
  • 17other files
View on GitHub