Skip to content

wtcooper/agent-evals

v0.1.0MIT

Evaluate an AI agent and the features switched on for it: guided reviews of computer use, memory, connectors, code execution and other agent features.

agent-feature-review

Guided security review of a new agent feature being switched on for a model or product the organisation already uses, when someone asks "can we enable this?" and no single benchmark answers it. One threat-and-test template per feature type: computer use and browser control, persistent memory, connectors and MCP servers, code execution sandboxes, file search and RAG, multi-agent and subagents, scheduled and background agents. Each template says what the feature changes about the trust boundary and, per threat, the review questions, the vendor evidence to look for, which existing benchmark could test it and through which skill, and what has no standardized test. Threats carry OWASP LLM 2025, OWASP Agentic and MITRE ATLAS ids where verified. Writes a JSON and markdown record, evidence.jsonl rows, and a list of every threat left unassessed. It prescribes no control and makes no recommendation. For an image-generation model use image-model-review.

License
MIT
Compatibility
Python 3.10+ standard library only. No API key and no network access are needed by the scripts.
Read SKILL.md at the source

Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.