Skip to content

zotoio/zoto-eval-system

v0.3.1MIT

Generate, run, and consciously update static (pytest) + LLM (@cursor/sdk) evals when AI primitives change — every proposed change requires user confirmation. YAML+JSONSchema results, surgical update on git diff, sovereign user-authored cases, /canvas-rendered cross-run compare.

zoto-compare-evals

Cross-run analysis across two or more eval runs driven by **report.yml**. Inputs are resolved per run directory under {evalsDir}/_runs/<ts>/; per-backend summaries are nested under **report.static** and **report.llm**. Emits a flat dataset (runs × cases × dimensions) plus the /canvas hand-off prompt, instructing the host agent to invoke Cursor's built-in /canvas tool. The skill never renders charts itself. Drill-down opens per-case log files. Does not call askQuestion — when a fragment (e.g. 20260503) matches multiple hourly folders and the Task lacks a disambiguator, emit schema-valid **needs_user_input** whose **options[].label** lists every full **{evalsDir}/_runs/<ts>/** candidate path (/z-eval-compare surfaces this without **askQuestion**).

Read SKILL.md at the source

Pinned to revision d79ebb356136, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.