design-llm-eval-suite
Designs and implements an offline eval suite for an LLM feature: golden, edge, and negative cases per prompt template, the cheapest scoring strategy that can actually fail, pinned determinism, a regression gate against a committed baseline, and the CI job that runs it. Use when an LLM feature ships without evals, when prompt or model changes keep causing silent regressions, or before switching models. For prompt injection and runtime guardrail testing, use the test-llm-guardrails command.
Pinned to revision c4efc46815a7, so it is the text this page describes rather than whatever the author pushed since.
Files
Every link opens the file at its source, pinned to the revision this page describes.