eval-ladder
Design and audit the eval ladder for an agent system: build tiers bottom-up from observed failures, pick the cheapest rung that can catch a given regression, name what each rung structurally cannot prove, validate every LLM judge against human labels (TPR and TNR separately, never raw agreement), and score irreversible-action scenarios pass^k rather than by majority. Use when designing, auditing, or defending a test/eval strategy for an agent, skill, or prompt; when adding an eval tier or LLM judge; when a suite is all-green and you cannot say what it would catch; or on phrases like "assess our testing strategy", "do we have eval coverage", "is this judge trustworthy", "pass@k or pass^k", "what does this tier actually prove".
