wtcooper/agent-evals
Evaluate an AI agent and the features switched on for it: guided reviews of computer use, memory, connectors, code execution and other agent features.
Guided security review of a new agent feature being switched on for a model or product the organisation already uses, when someone asks "can we enable this?" and no single benchmark answers it. One threat-and-test template per feature type: computer use and browser control, persistent memory, connectors and MCP servers, code execution sandboxes, file search and RAG, multi-agent and subagents, scheduled and background agents. Each template says what the feature changes about the trust boundary and, per threat, the review questions, the vendor evidence to look for, which existing benchmark could test it and through which skill, and what has no standardized test. Threats carry OWASP LLM 2025, OWASP Agentic and MITRE ATLAS ids where verified. Writes a JSON and markdown record, evidence.jsonl rows, and a list of every threat left unassessed. It prescribes no control and makes no recommendation. For an image-generation model use image-model-review.
SCAFFOLD, not a finished benchmark. Measures whether a model running inside an agent harness (a coding agent CLI, an IDE agent, a vendor's own harness) carries out tasks an organisation's policy says it must decline, such as offensive security work, and whether it still completes the matched legitimate tasks. The answer depends on the harness and its safeguards as well as the model, so the arms are model × harness × safeguards. This skill ships the scenario format, the arm sweep, the statistics and the report, with a mock for a keyless smoke test. It ships no scenarios, no real harness and no outcome scorer: the team using it supplies those, and the build guide says how. Use it when you are ready to build that for your own policy.