agent-misuse-benchmark
SCAFFOLD, not a finished benchmark. Measures whether a model running inside an agent harness (a coding agent CLI, an IDE agent, a vendor's own harness) carries out tasks an organisation's policy says it must decline, such as offensive security work, and whether it still completes the matched legitimate tasks. The answer depends on the harness and its safeguards as well as the model, so the arms are model × harness × safeguards. This skill ships the scenario format, the arm sweep, the statistics and the report, with a mock for a keyless smoke test. It ships no scenarios, no real harness and no outcome scorer: the team using it supplies those, and the build guide says how. Use it when you are ready to build that for your own policy.
- License
- MIT
- Compatibility
- The scaffold needs Python 3.11+ only. A completed benchmark needs whatever your harness module needs, typically Docker and inspect_ai with inspect-swe.
Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/agent-misuse-benchmark/SKILL.md
- skills/agent-misuse-benchmark/interfaces.py
- skills/agent-misuse-benchmark/lib/evalkit.py
- skills/agent-misuse-benchmark/metrics.py
- skills/agent-misuse-benchmark/new_experiment.sh
- skills/agent-misuse-benchmark/references/build-guide.md
- skills/agent-misuse-benchmark/references/scenario-format.md
- skills/agent-misuse-benchmark/run_benchmark.py
- skills/agent-misuse-benchmark/scenarios.example/arms.mock.json
- skills/agent-misuse-benchmark/scenarios.example/pack.json
- skills/agent-misuse-benchmark/scenarios.py
- skills/agent-misuse-benchmark/tests/test_agent_misuse_scaffold.py
- skills/agent-misuse-benchmark/validate_scenarios.py
Every link opens the file at its source, pinned to the revision this page describes.