zotoio/zoto-eval-system

Coverage gap analyser for eval suites. Scans the codebase across five dimensions — trigger-phrase coverage, schema validation, regression baselines, context citation verification, and status checklist completeness — and produces a structured adviser_report with per-target, per-dimension findings and deterministic recommendations. Read-only: never modifies files. Does not call askQuestion — uses needs_user_input at two breakpoints (summary drill-down and action recommendations) for the command to drive via askQuestion and resume.

Cross-run analysis across two or more eval runs driven by **report.yml**. Inputs are resolved per run directory under {evalsDir}/_runs/<ts>/; per-backend summaries are nested under **report.static** and **report.llm**. Emits a flat dataset (runs × cases × dimensions) plus the /canvas hand-off prompt, instructing the host agent to invoke Cursor's built-in /canvas tool. The skill never renders charts itself. Drill-down opens per-case log files. Does not call askQuestion — when a fragment (e.g. 20260503) matches multiple hourly folders and the Task lacks a disambiguator, emit schema-valid **needs_user_input** whose **options[].label** lists every full **{evalsDir}/_runs/<ts>/** candidate path (/z-eval-compare surfaces this without **askQuestion**).

Writes .zoto/eval-system/config.yml from field values pre-collected by /z-eval-configure (command-owned askQuestion). Refuses bundled update.preserveUserAuthoredCases:false or update.writeMetaMarker:false (including nested payload mirrors) before any manifest read or config write. Validates against templates/schema/config.schema.json. Diffs the chosen config against the manifest snapshot (discovery_config.static.framework) to produce a cleanup_plan validated against templates/schema/cleanup-plan.schema.json, and stamps _meta.primitive_analysis.invalidate=true on every cached primitive analysis when the static framework changes. The unified LLM eval harness ships one JSON-first co-located eval per target, with no per-repo selector axis — only the static framework matters here. Never calls askQuestion — returns needs_user_input if validation cannot proceed without user clarification.

Scaffolds static pytest and LLM (@cursor/sdk) eval backends into a host repository, generates eval cases with _meta.generated markers for approved skills/commands/agents/hooks, merges eval scripts into package.json, and writes the persistent manifest plus append-only history. Expects the invoking command to pre-collect approved targets — does not call askQuestion.

Invocation reference for all eval-system CLI commands. Use this skill when you need to run eval scripts — never reference script paths directly; always invoke via the package.json aliases documented here.

Executes eval runs via the host repo's package.json scripts (pnpm run eval, eval:full, eval:llm, eval:judge). Forwards --model to the LLM runner via both a CLI flag and the ZOTO_EVAL_MODEL environment variable. After each run, calls eval:update --check and appends the drift status to the run report as a warn-only line. Does not call askQuestion — credential gaps use needs_user_input for the command.

Project-tailored help for the Eval System plugin. Loads plugins/zoto-eval-system/README.md as the source of truth, inspects current project state (.zoto/eval-system/config.yml, manifest.yml, .env*, evals/_runs/, package.json scripts), and composes an answer that is grounded in the chosen README section, tailored to the host repo's actual configuration and artefacts, and cited with start:end:path code references back to the README. Pre-collected help_context arrives from the /z-eval-help command (command-owned askQuestion). The skill itself never calls askQuestion — missing context is returned as needs_user_input.

Adversarial judge for eval runs. Identifies weak graders, under-covered assertions, verbosity spikes, and accuracy/confidence regressions. Produces an enriched llm.yml with soft-metric annotations. Does not call askQuestion — handoff to /z-eval-update is expressed as needs_user_input for the command to confirm via askQuestion and resume. Uses the configured judgeModel (default claude-opus-4-8[]).

Diff-aware eval updater. Detects drift between covered targets and generated eval cases for skills and central plugin eval files (commands/agents/hooks), classifies changes as critical or non-critical, and preserves user-authored cases verbatim. Apply-mode regeneration re-invokes the LLM analyser per drifted primitive and dispatches per-framework static stamping plus the single unified LLM regeneration (regenerateLlm) (CI defaults to cached analyser payloads unless --with-analyser). The _meta.generated === true (case) and // _meta.generated\: true (file) contracts are enforced at runtime AND compile time. Supports targeted mode, rediscovery mode, --no-analyser / CI cache reuse, and CI --check (exit 2 on critical drift).