agent-evals

Use this skill when a Copilot skill, prompt, or agent changes and someone needs evidence that it works rather than one lucky run. Covers three stacked questions: is it discovered (trigger-rate evals), does it hold up across runs (capability and regression sets, pass@k vs pass^k, deterministic / LLM-as-judge / human graders), and does it beat no skill at all (with/without delta on pass rate, tokens, duration). Starts from native VS Code tooling and Waza, falling back to bundled PowerShell harnesses. Rule of thumb: start from 20-50 real failures, not synthetic prompts. USE FOR: evaluating or measuring any Customization, eval harness and artifact design, grader selection, trigger-rate and discoverability work, eval-driven iteration on a description or a body. DO NOT USE FOR: authoring the skill itself (use skill-creator), MCP server eval questions (use mcp-builder Phase 4), unit-testing PowerShell code (use pester-patterns), security review of an agent (use agent-security-review).

Compatibility

The bundled harnesses need PowerShell 7. Trigger Prepare and Execute also need powershell-yaml; Execute needs ShellPilot plus a paid model backend. Grade needs neither powershell-yaml nor a backend. The default -Dispatch Batch needs 0.4.0-preview0005 for Invoke-ShpBatch, -Dispatch Sequential runs on 0.4.0, and -Temperature needs preview0004 - a version test cannot decide that, because preview0003 reports 0.4.0 and rejects it.