Skip to content

wtcooper/agent-evals

v0.1.0MIT

Evaluate an AI agent and the features switched on for it: guided reviews of computer use, memory, connectors, code execution and other agent features.

agent-misuse-benchmark

SCAFFOLD, not a finished benchmark. Measures whether a model running inside an agent harness (a coding agent CLI, an IDE agent, a vendor's own harness) carries out tasks an organisation's policy says it must decline, such as offensive security work, and whether it still completes the matched legitimate tasks. The answer depends on the harness and its safeguards as well as the model, so the arms are model × harness × safeguards. This skill ships the scenario format, the arm sweep, the statistics and the report, with a mock for a keyless smoke test. It ships no scenarios, no real harness and no outcome scorer: the team using it supplies those, and the build guide says how. Use it when you are ready to build that for your own policy.

License
MIT
Compatibility
The scaffold needs Python 3.11+ only. A completed benchmark needs whatever your harness module needs, typically Docker and inspect_ai with inspect-swe.
Read SKILL.md at the source

Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.