jaktestowac/ai-quality

Designs and implements an offline eval suite for an LLM feature: golden, edge, and negative cases per prompt template, the cheapest scoring strategy that can actually fail, pinned determinism, a regression gate against a committed baseline, and the CI job that runs it. Use when an LLM feature ships without evals, when prompt or model changes keep causing silent regressions, or before switching models. For prompt injection and runtime guardrail testing, use the test-llm-guardrails command.

Human review protocol for whether AI output is grounded in its sources: claim-by-claim attribution, hallucination classes, citation verification, sampling that is defensible, and a recorded attestation because groundedness cannot be fully automated. Also covers the responsible-AI pass - harmful output, bias, disclosure and privacy. Use when checking whether RAG answers or summaries are supported by the retrieved documents, when factuality matters more than fluency, or when an eval suite needs the human check it cannot replace.

Maps the trust boundaries of an LLM feature, tests the runtime guardrails around model output (schema, PII, moderation, rendering, tool authorization, resource caps, fail-closed behaviour), and runs an adversarial corpus through every content channel, including retrieval and tool results. Use when an LLM feature reads untrusted content, calls tools, or renders model output, and before such a feature ships. Runs only after written authorization for a named environment. For output-quality regression evals, use the design-llm-eval-suite command.

Tests the customization assets themselves - skills, prompts, custom agents, instructions - the way a product is tested: activation cases that check an asset activates when it should and stays quiet when it should not, output-contract cases, safety cases, collision cases between assets competing for the same trigger, a weighted rubric scored blind, and a gate comparing the edited version against the previous one before it ships. Use when a skill is edited and nobody knows whether behaviour changed, when two skills fight over the same request, when a description is being tuned for discoverability, when a collection has grown past manual spot-checking, or when the request mentions skill evals, prompt regression, or "does this skill actually work".

Builds an offline eval suite for LLM-powered features: golden reference cases per prompt template, deterministic scoring where possible, a regression gate on prompt, model and retrieval changes, and CI wiring with promptfoo, Vitest or deepeval. Use when a product ships prompts, agents or RAG, when a model or prompt upgrade needs a regression check, when LLM output is currently verified by someone eyeballing it, or when asked how to test a feature whose output is non-deterministic.

Tests the runtime validation around model output - schema conformance, PII leakage, moderation, refusal behaviour, tool-call safety - and the adversarial suite that tries to defeat it: direct prompt injection, jailbreaks, and indirect injection through retrieved documents, tool results and user content. Use when an LLM feature reaches real users, when a model can call tools or read untrusted content, when an AI feature needs a security-style test suite, or when asked whether a chatbot or agent can be manipulated.

Cuts AI tells from what the agent says about its own work: completion claimed without a run, invented file paths and API names, terminal output that was never produced, findings with no file or line, percentages with no denominator, absence claimed from one grep, effort narration instead of results, hedge stacking, buried blockers, puffery in a bug title. Use whenever reporting a result, a fix, a review finding, a test outcome, a coverage or flake number, a root cause, or a release recommendation, and when the request mentions "prove it", "did you actually run it", "be specific", "no fluff", "stop hedging", or "is this real". Must always apply to answers about your own work.