skill-tuner

Tells you whether a change to an agent-consumed document actually worked — with an interval, a margin you set, and a receipt that can be re-checked after the model moves under you.

It does not help you write skills. Plenty of guides do that. What nothing does is tell you whether the edit you just made was an improvement or a coin flip.

The problem, with receipts

Eval harnesses report deltas. Anthropic's own skill-creator runs a proper experiment — eval cases, an isolated subagent per run, graded expectations, with_skill vs without_skill benchmarking — and aggregate_benchmark.py reports mean and stddev. Its improvement record stores a bare grading_result: "won" | "lost" | "tie" beside a pass rate. Claude Code's own claude plugin eval (v2.1.269, 2026-09) does the same for any plugin: three runs per case, a meanDelta, and a pass/fail against --threshold — no interval, no margin, and no way to compare two versions of the same plugin.

A threshold with no interval behind it cannot say we could not tell, so it answers every time — and on a noisy measurement most of those answers are luck.

We know because it happened to us. One question — is this doctrine better than the one it forked from? — asked three times with a raw-count rule:

Two of three were wrong. The same rule produced both. That cost ~$30 and a day to discover, and it is the single reason this project exists in its current form.

What it does

tune.py compare --skill-creator <benchmark-dir> \
  --baseline without_skill --candidate with_skill --delta 0.05

A claude plugin eval run is read the same way, straight from its aggregate-result.json (the arms of one run, or two runs of the same suite by case):

tune.py compare --plugin-eval \
  --baseline evals/results/<run>/aggregate-result.json@without \
  --candidate evals/results/<run>/aggregate-result.json@with --delta 0.1

The same verdict engine accepts any external paired series — two flat {case: number} JSON files from whatever produced your scores (e.g. memory-dream's eval export-paired):

tune.py compare --paired-json \
  --baseline before-scores.json --candidate after-scores.json --delta 2.0

--delta is always yours to state in that mode: the margin is in the units of your series, and a tool that guessed it would be repeating the founding mistake above.

# Paired comparison — skill-creator benchmark

**Verdict: inconclusive**  (non-inferiority margin delta = 0.05 pass_rate per case)

- pass_rate paired across 6 cases (with_skill vs without_skill)
- mean difference: +0.017 per case (sd 0.279, se 0.114)
- 95% CI: [-0.28, +0.31]
- case record: 3 won, 2 lost, 1 tied
- a bare total comparison would say: PASS
- to clear the margin at this mean and spread you would need about 68 cases (+62 on this run)

On that benchmark with_skill averages 0.550 against 0.533. A total comparison calls it a win. It is not distinguishable from zero, and you would need about 68 cases before it could be.

Three things, none of which a first-party harness gives you (skill-creator and claude plugin eval both stop at the delta):

• A verdict, not a delta — paired by case, 95% interval, a non-inferiority margin you state, and four outcomes including inconclusive. A percentile bootstrap, an exact sign test, and a paired effect size print beside the t-interval, so a verdict leaning on a normality assumption is visible too. It prints what a bare comparison would have concluded next to its own, so the difference is visible rather than argued.
• Provenance — every input content-hashed and git-pinned, model and CLI version recorded. tune.py verify <run> re-checks a finished run for $0 and tells you what drifted. It caught a real input change within hours of being written.
• An adversarial probe — applies an authoring doctrine to a document, judges each finding with an independent skeptic panel, and downgrades any finding whose quote is not actually in the target, regardless of how the panel voted.

Install

Two steps, from inside Claude Code:

/plugin marketplace add https://github.com/StartupBros-com/hov-marketplace.git
/plugin install skill-tuner@hov

The runner shells out to the claude CLI, so that needs to be installed too. Every call runs with CLAUDE_CODE_DISABLE_CLAUDE_MDS=1, so your CLAUDE.md, project instructions and auto-memory never reach an arm (before v0.8.2 they did: a confound, and about ten times the per-call cost, measured in docs/COSTS.md). Your login is untouched, which is why the runner does not use --bare.

Using it

/skill-tuner:tune <path> — point it at a SKILL.md, AGENTS.md or CLAUDE.md and your agent will probe it, fix what the probe confirms, re-probe until those findings are gone, and prove the description still routes if it changed. The loop ships as the user-invoked tune skill (skills/tune/SKILL.md), so clients that load Agent Skills from plugins — Codex among them — surface it too; the runner itself still needs the claude CLI installed.

Findings arrive pre-verified — three independent skeptics in fresh contexts, plus a code-level check that the quoted text actually appears in the document — so the agent acts on them rather than re-litigating them. Refuted findings are shown too, labelled with what killed them.

The command is deliberately explicit about what it may not claim: a before/after count on one document is not evidence of improvement. For that you need a paired comparison across many documents, which is what tune.py compare is for.

The runner never fires on the model's own initiative. It spends your tokens, so it starts on your instruction, and reports the estimate before the first call and the actual after.

/skill-tuner:portfolio [project dir] — the same question across every skill at once. tune.py portfolio (no model calls, reads files only) lists each skill and command the model is shown, from your skills and commands folders, claude.ai-synced skills, the project and every enabled plugin, with its per-turn listing cost, its usage count, the setting that decides its visibility, and a per-plugin rollup of what a project-level enabledPlugins: false would remove. The skill then walks the costliest ones with you one at a time, asking first whether the model should start each skill on its own, and applies only what you choose through the knob that governs it (frontmatter, skillOverrides, or, for plugin skills, which have no per-skill knob, the per-project plugin switch). Usage counts inform the walk and never decide it: a skill nobody triggers may need a better description, not hiding. It reports what a turn pays before and after (over budget, the most the model can see, not what the skills demand), and claims nothing about routing or quality.

For a one-off, without the loop:

python3 skills/skill-tuner/scripts/tune.py probe --target path/to/SKILL.md --yes --budget-usd 3 --verify-trials 3

Works with skill-creator and claude plugin eval, not against them

They run the experiment; skill-tuner decides what it means. Integration is at the file boundary — it reads grading.json / the benchmark tree, or aggregate-result.json, and never imports their code, because their scripts have no API contract and the marketplace bumps plugin SHAs nightly. aggregate-result.json is the friendlier input: a versioned schema with per-case, per-arm scores.

Use theirs for eval cases, grading, trigger tuning and benchmarking. Use this for the verdict. This repo ships its own evals/ suite (two cases: a natural-phrasing request graded twice, once for whether the skill fired and once for the doctrine's own vocabulary in the answer, and an unrelated-request control); claude plugin eval . from the plugin root runs it, about $0.50 at one run per arm. Its results feed compare --plugin-eval, which refuses a partial document, a case list that disagrees with the document's own casesTotal, and a case with an errored run in the requested arm (unless you --exclude it).

One routing caveat measured while building it: a skill's name routes on its own. With the description replaced by Internal notes., skill-tuner still fired 3/3 on a "fix my SKILL.md" prompt; with a neutral name the same change went 0/3 to 3/3. The routing-parity battery shows the id beside the description, so neutral_ids: true in a config hides it. Re-running the receipts battery that way (reports/route-neutral-20260920/, $2.60): original descriptions 26/26, pruned 23/26, near-miss rejection 8/8 both, verdict refuse — the same verdict and the same failing skill (agent-swarm) as the id-visible run (reports/receipts-routing-001/, 24/26 vs 23/26). Hiding the id did not rescue the prune, and it did not hurt the originals; the 2026-08-06 30/30 run predates the option and has not been re-run under it.

What this does not measure

Two axes are out of scope, and the ecosystem's loudest 2026 numbers live on them. Whether a skill makes an agent better at its end task is skill-creator's benchmark question (and SkillsBench's, at academic scale) — this tool reads those results, it does not produce them. Whether a skill is safe to install — prompt injection, exfiltration — is a scanner's question (Snyk's ToxicSkills audit found injection in 36% of 22,511 public skills). skill-tuner grades the authoring and gates the edit: point it at a document you already trust, beside a benchmark you already run.

Case studies on the public record

Start with the August 2026 campaign — the tool pointed at its own harness and then at corpora we didn't write: 96 verified fixes at certified behavioral parity, two expert-endorsed doctrine changes refused by measurement, a cross-family judge audit, and a CI-backed 61% defect reduction on external OSS skills, with upstream PRs to anthropics/skills among the receipts.

The verdict engine applied to the ecosystem's own disputes, at zero model cost: the Ponytail benchmark saga — the famous "54% less code" is a totals artifact; the per-task paired truth is 35.4% [15, 56], direction near-certain, wildly task-dependent — and SkillOpt's strictly-improves gate, whose per-edit re-analysis is fully specified and blocked only on data its authors state is not public.

Receipts

Live runs against claude -p, reproducible from committed configs. The routing eval refused a description prune that lost accuracy; the probe confirmed 19 defects across sixteen already-shipped documents. A gate that only ever says yes is not a gate, so both outcomes are published.

Routing-parity — receipts-routing-001

Verdict: refuse

Failing case ids: agent-swarm__obvious__2, agent-swarm__paraphrase__1

Marginal-value probe — swapgate3-probe-candidate

findings_confirmed: 19 (refuted: 32, targets: 16, probe calls: 16, verify calls: 153 at 3 skeptics per finding)

• Single source of truth — duplication of one meaning across sections: restates, nearly verbatim, a rule already given under 'Pick the lightest engine that fits'…
• Leading words — a coined term must be defined, not assumed: 'Doghouse' is used as if it were a known term but is never defined anywhere in the document…
• Completion criteria [craft]: this checklist item gives no checkable bound — 'the perception threshold' names no value or method to test against…
• …and 16 more, with quotes and proposed fixes, in reports/swapgate3-probe-candidate/report.md

Reproduce:

python3 skills/skill-tuner/scripts/tune.py probe --config configs/swapgate5-probe-doctrine-v2.json --run-id my-run --yes --budget-usd 11
python3 skills/skill-tuner/scripts/tune.py verify my-run

Session-measured on the authors' harness (2026-08-08). Costs and what actually reduces them are measured in docs/COSTS.md — including two optimizations we found and deliberately did not switch on, because they change the prompt envelope and would invalidate every banked baseline.

The doctrine is the worked example, not the product

skills/skill-tuner/SKILL.md is an authoring doctrine where every rule carries an evidence tag and a falsifier — and, new in v3, every [research] tag carries a fetched-and-verified citation in FALSIFIERS.md. It descends from mattpocock/skills' writing-for-agents (MIT). The incumbent it is measured against was frozen from upstream on 2026-08-05 and includes upstream's 2026-07-23 rename/restructure and its last content change (2026-07-28), plus one local sentence this project added (the time-relative no-op amendment) — so the comparison is against current upstream content, made strictly harder by that amendment. (An earlier revision of this paragraph called the snapshot stale; checking upstream's commit log disproved that, and the correction is this sentence.)

Measured on 15 shared documents, four times, against the same banked ancestor leg:

The v3.1 row is the strongest advertisement here: four sentences restored from the ancestor, each endorsed by a 13-reviewer audit as worth having, cost 43% of the doctrine's yield on identical documents (reports/swapgate7-summary.md). Every expert in the pipeline said yes; the measurement said no; v3 ships.

† The re-baseline replication (reports/rebank-sys-summary.md): new prompt envelope, and a corpus in which 51 verified defects — disproportionately v3's favorite duplication class — had since been fixed. The near-boundary better did not survive that; the direction did, in 9 of 15 documents. The standing claim across every measurement: v3 ≥ its ancestor everywhere measured (defect-finding twice, authoring once), strictly better once, and this table publishes the whole trajectory rather than the best row.

* v1 was originally published as worse. A 2026-08-08 statistics review found the worse verdict anchored at zero rather than −δ, so it fired on any confirmed regression even inside the declared-tolerable band; the corrected reading of the same interval is a confirmed regression of unresolved size against the margin. The data did not move — the label did, and the fix is in compare.py with a regression test.

Three gates in a row said not better — and every one shipped, because publishing the losses is the point. The fourth (swapgate6, 2026-08-08) is the first better, and it gets the same calibration as the losses did: the interval clears zero by +0.03, the bootstrap CI agrees, the sign test alone would not resolve it — a near-boundary win, not a landslide. The rewrite behind it did not chase style. A forensic diff of the earlier legs located the deficit in one place — duplication-hunting was 41% of the ancestor's confirmed yield and 4% of v2's — and v3 restored that single rule to the terminal, elaborated position it holds in the ancestor. The category came back 1 → 16, and the four gates move together: 18 → 22 → 27 → 42 confirmed, mean −0.73 → −0.47 → −0.13 → +0.87 (reports/swapgate6-summary.md).

If you want a doctrine, use either. If you want to know whether the one you picked is doing anything, that's what the rest of this repo is for.

Attribution

Doctrine descends from writing-for-agents (MIT). Reads benchmark output from Anthropic's skill-creator (Apache 2.0), which it complements rather than replaces.