danielsimonjr/verify-mcp
MCP server wrapping danielsimonjr/verify (veriharness).
Changelog
All notable changes to this project are recorded here. The format follows Keep a Changelog, and the project uses Semantic Versioning.
[Unreleased]
[0.16.12] - 2026-10-12
Changed
- The pin moves to verify 0.12.15. The warning for a window at or below 16384 tokens says that every
tool result is cut to 1000 characters, which is what a session measured at that size does; it no
longer says that every rollout compacts. The floor for
context_sizeis the same: above 4096.
[0.16.11] - 2026-10-12
Changed
- The pin moves to verify 0.12.14. A pi worker whose final message is a tool call written as text (prose
before the call, or a value that is not JSON) is sent back once, on a turn that offers the tools, with
the reason that the call did not run. The guard cuts a message that writes a second such call. The
nudge kinds of
nudge_timeoutare now time, cut, stuck, format, json, schema, ids and call.
[0.16.10] - 2026-10-12
Changed
- The pin moves to verify 0.12.13. A pi worker is stopped when a tool call holds the answer records (the
error
stuck, then the nudge with no tools), a call with an argument that the tool does not list is blocked with the arguments it takes, a message that the guard cut is sent back with the reason, and a final message with broken JSON is named at the place where it breaks in every case.
[0.16.9] - 2026-10-12
Changed
- The pin moves to verify 0.12.12. A pi worker is stopped after three turns in which every tool call
failed (the error
stuck), each tool result is cut to the room that is left before compaction, a final message with broken JSON is sent back with the place where it breaks, and withidsthe output limit follows the size of the batch.retriesalso runs astuckor acompactedrollout again.
[0.16.8] - 2026-10-12
Added
verify_workerstakesfrequency_penalty(passed as--frequency-penalty, verify 0.12.11 or later). It is not set by default.
Changed
- The pin moves to verify 0.12.11. A pi worker loads a repeat guard that blocks a repeated read, grep,
find or ls call and cuts a final message that has more records than the batch has items. A pi worker
is sampled with temperature 0.6 when
temperatureis not set. Thetemperaturedescription says so.
[0.16.7] - 2026-10-11
Changed
- The pin moves to verify 0.12.10. A pi worker is sent back for its answer once for each kind of
failure (time, cut, format, schema, ids) instead of once in all, and with
idsits charter lists the ids of the batch.nudge_timeoutis the budget of each nudge.
[0.16.6] - 2026-10-11
Added
verify_workerstakesids(ARRAY.FIELD, passed as--ids). A report that fits the schema and covers other rows than its batch ends as errorids, is sent back once, and is retried byretries.
Changed
- The pin moves to verify 0.12.9. A pi worker that made no tool call ends as the new error
no-tool-callsinstead of complete, and the schema is in the worker charter.
[0.16.5] - 2026-10-11
Changed
- The pin moves to verify 0.12.5.
driver.loglines carry the local time and its UTC offset, and the manifest, eachworker.jsonandresult.jsonname the provider and model of every role, also a Claude Code model.
[0.16.4] - 2026-10-10
Changed
- The pinned
verifyis 0.12.4.verify_model_check,verify_batch,verify_workersandverify_drivernow report the digest and quantization of a local model, so a re-created tag shows in the records.
[0.16.3] - 2026-10-10
Changed
- The verify pin is 0.12.3:
verify_workersnames an--item-tokensvalue only for a peak that is past the budget of the manifest, since a rebatch with a smaller value gave the same split.
[0.16.2] - 2026-10-10
Changed
- The verify pin is 0.12.2: the driver line for an investigation that left no record names the compactions of its session.
[0.16.1] - 2026-10-10
Changed
- The verify pin is 0.12.1: the
workerswarning about failed tool calls names the class of error (toolErrorKinds) and no longer says the worker reads missing paths when it sent no arguments.
Fixed
- Two tests that start a process cold (
captures stdout and stderr, and the fresh-build check of the committed bundle) now carry an explicit 30 s test timeout. Bun's default of 5 s cut them short on a loaded or freshly scanned disk, so the suite failed once right afterbun installand then passed.
[0.16.0] - 2026-10-10
Changed
- The verify pin is 0.12.0.
verify_batchtext names eachworkspace/<file>that the spec or the worker prompt names and no batch holds (manifest.jsonmissing), withitems_nameas the usual fix.
[0.15.0] - 2026-10-10
Changed
- The verify pin is 0.11.0.
verify_drivertext lists the items the reviewer left open (result.jsonopenItems): the delivered file holds a verdict for each of them.verify_workerstext showsN s in allfor a retried rollout (worker.jsontotalSeconds), beside the seconds of the last attempt.
[0.14.1] - 2026-10-10
Changed
- The verify pin is 0.10.1: a record that an investigation wrote in its own folder is no longer
judged missing, and the records of an earlier run are set aside to
previous/(verify CHANGELOG 0.10.1).
[0.14.0] - 2026-10-10
Changed
- The verify pin is 0.10.0.
verify_workersnames the outcome of each rollout in its result text (b01/r01: max-turns (60 s, 41 turns, peak 28881)) and repeats the harness warnings. A host that shows only the text of an error result no longer loses which rollout failed and why.verify_driverreturns the driver'sresult.json(verify 0.10.0 or later) asstructured.resultand states in its text which investigations left no record, the base (nonemeans the deliverable was built from the inputs), an invalid delivery, and any write an investigation made outside its own files. Exit 0 says only that the run ended.
[0.13.0] - 2026-10-10
Added
verify_workerstakesschema,retries,nudge_timeoutandallow_compaction. They pass--schema,--retries,--nudge-timeoutand--allow-compactionto verify 0.9.0. A deliverable that does not fit the schema ends asschema; a stopped pi worker gets one nudge for its answer; a rollout with a compaction ends ascompactedunlessallow_compactionis true.
Changed
- The verify pin is 0.9.0.
[0.12.0] - 2026-10-10
Added
VERIFY_MCP_ITEM_TOKENSandVERIFY_MCP_OVERHEAD_TOKENSset the defaultitem_tokensandoverhead_tokensofverify_batch, in the serverenvofsettings.jsonor.mcp.json. What one item costs a worker depends on the task, the tools and the model, so a deployment sets its own measured value and no default is built in. A call that sets its own value keeps it. The variables apply whether or not the call names a model.
Changed
- The verify pin is 0.8.0. A
verify_workersrollout record now also hasestTokens, the batch estimate frommanifest.json, and verify writes a progress line when the measuredpeakContextis above it. The line names the--item-tokensvalue that would have covered the batch. verify-mcp passes the record through unchanged.
[0.11.0] - 2026-10-09
Added
verify_batchtakesreference, passed as--reference: files every batch gets inworkspace/that the worker only searches, so the budget does not count them.
Changed
- The verify pin is 0.7.0, the first version with
veriharness batch --reference.
Fixed
- The README's
verify_batchexample passed the searched CHANGELOG asshared, which puts every batch over the budget. It now usesreference.
[0.10.0] - 2026-10-09
Added
verify_workerstakesmax_turns, passed as--max-turns: a worker that passes that number of assistant turns stops with the errormax-turns.
Changed
- The verify pin is 0.6.0, the first version with
veriharness workers --max-turns.
[0.9.0] - 2026-10-09
Added
verify_batch: runsveriharness batch. It splits an items file (jsonl,blank-lineorheading:REGEX) into task folders that each fit a token budget,batch_tokensor half the worker model's window. The result holds the parsedmanifest.json. The default model profile applies when the call sets nobatch_tokens.verify_workers: runsveriharness workers, N worker rollouts on each batch. Each finished rollout is one progress notice, sent at once. The result holdsrolloutsandsummary; exit 1 and exit 75 (a usage limit) are errors that still carry the records. Default timeout 12 hours.context_size: "auto"on the main model and on each role. A role,verify_batchandverify_workerstake"auto"or a whole number above 4096, as verify does.verify_model_checkreturnswindowandwindowSourcefrom verify.
Changed
- Pin verify 0.5.0, which adds
--context-size auto,batchandworkers.
[0.8.0] - 2026-10-09
Added
rolesonverify_driverandverify_runner: a strict object keyed bychecker,challenger,reviewerandfixer. Each entry is{provider, model, base_url?, context_size?}and becomes--role ROLE=PROVIDER:MODEL, plus--role-base-urland--role-context-sizefor a local role. README section "A model for each role"; the skill names the field.
Changed
- Pin verify 0.4.0, which adds the role options.
tests/roles.test.tskeepsROLESequal to the pinnedharness/roles.ts.
[0.7.0] - 2026-10-09
Changed
- The pin moves to verify 0.3.0, which renames the lane
flashtofable.LANESand thelane_maxkeys follow, andmax_flashbecomesmax_fable. A cell on an archivedflashpool still runs: verify gives it thefablelane. - README: the four phases carry the verify role names Checker, Challenger, Reviewer and Fixer. The Worker makes each rollout. Tool names, fields and files are unchanged.
[0.6.0] - 2026-10-09
Changed
- The pin moves to verify 0.2.0. All four lanes (
flash,opus,haiku,sonnet) run Claude Code, so every lane needsenv: "none". The text of the tool schemas, the README and the verify skill says so. - verify sets the default lane caps: flash 2, opus 2, haiku 4, sonnet 4. The wrapper default of 4 for haiku and sonnet from 0.5.1 is removed, because the pinned verify now holds the same value.
[0.5.1] - 2026-10-09
Changed
verify_runnerraises the in-flight cap of the haiku and sonnet lanes from verify's default of 2 to 4. The server adds--lane-max haiku=4or--lane-max sonnet=4only when the cells use that lane andlane_maxdoes not name it. A cap that the call sets still wins. A run that uses no Claude Code lane passes no new flag.
[0.5.0] - 2026-10-07
Added
- Tuning in the default profile:
VERIFY_MCP_THINKINGandVERIFY_MCP_NUDGE_TIMEOUT(verify_driver), andVERIFY_MCP_MAX_TOKENSandVERIFY_MCP_REQUEST_TIMEOUT(verify_driverandverify_model_check). They follow the same rule as the other profile fields: used only when the call names no provider and no model, and only for a field the call omits.plugin/.mcp.jsonpasses them through empty andexample.mcp.jsonlists them. example.mcp.json: a server entry with everyVERIFY_MCP_*variable and placeholder values, for a user to copy and set after installing. The README describes it. Git ignores a root.mcp.json, so a local copy stays local.
[0.4.2] - 2026-10-07
Changed
- The plugin ships no default model. 0.4.0 and 0.4.1 set one in
plugin/.mcp.json, which tied a local setup to every install.plugin/.mcp.jsonnow passes the fiveVERIFY_MCP_*variables through, empty when unset, and a test pins this. A machine sets its own default in theenvblock of its Claude Codesettings.json.
[0.4.1] - 2026-10-07
Changed
- The plugin default model changed to a variant that carries a
num_ctx, because verify refuses a model that reports none. Removed in 0.4.2.
[0.4.0] - 2026-10-07
Added
- A default model profile from the environment:
VERIFY_MCP_PROVIDER,VERIFY_MCP_MODEL,VERIFY_MCP_BASE_URL,VERIFY_MCP_CONTEXT_SIZEandVERIFY_MCP_ENV.verify_driverandverify_model_checkuse it when a call names no provider and no model.
Changed
verify_model_checkno longer requiresproviderandmodelin its schema. It returns an error that names the variables when neither the call nor the environment gives them.
[0.3.0] - 2026-10-06
Changed
- verify-mcp installs verify from npm: the dependency is
@danielsimonjr/verifyat exactly 0.1.0, and not a GitHub commit.src/pin.tsholds the package name and the version, and a test keepspackage.jsonequal to them.verify_statusreports the pin as@danielsimonjr/verify@0.1.0. - The plugin still starts the verify checkout through
VERIHARNESS_BIN: an installed plugin has nonode_modules.
[0.2.2] - 2026-10-06
Changed
- verify-mcp pins verify
74a39c5. That verify commit installs its pi agent runtime from@danielsimonjr/pi(0.84.4) and not from@earendil-works/pi-coding-agent. The plugin still launches the verify checkout throughVERIHARNESS_BIN, so a checkout needssetup_pi.shrun again after it is updated to that commit. No npm publish is involved: verify is a git dependency. - The README names the real pinned commit. It still named
0a7bbf3.
[0.2.1] - 2026-10-05
Changed
- verify-mcp pins verify
c10100b(verify PR #19). The dependency is namedverify, the package's new name, and the server looks fornode_modules/verify. The command and theVERIHARNESS_variables keep their names. - The new pin also includes verify PR #17 and PR #18. In a grader, the stdout cap now kills the whole process tree and runs the cleanup hook, as the timeout does. The live Claude Code test now checks that the verifier picks the right answer.
[0.2.0] - 2026-10-05
Added
- The Claude Code verifier. verify-mcp pins verify
0a7bbf3(verify PR #16). Since verify PR #11 (102894a), verify runs the verifier on Claude Haiku or Claude Sonnet throughclaude -pwith the login that Claude Code holds. The provider fields nameclaude-code.verify_model_checkprobes it. verify_runneraccepts the laneshaikuandsonnet,lane_max(one--lane-max LANE=Nper lane; a strict object, so an unknown lane or__proto__is an input error) andenv, which verify passes to every driver. The Claude Code lanes needenv: "none".- A
verify_model_checkerror for a verify build without the Claude Code provider says so and names the commit that added it. - The README, the
verifyskill and the/verifycommand describe the Claude Code verifier, its lanes, its Windows setup (VERIHARNESS_CLAUDE_BIN) and the fact that it always runs without the jail.
Fixed
- The text for a verify build without
model-checknamed the pinned commit as the commit that added it. It names756bc2b(verify PR #2) now; the pin moves and that commit does not. - The detached-grandchild test stops the run when the grandchild has started, not after a fixed
1.5 s timeout. Two Bun cold starts can take longer than 1.5 s on a loaded host; the stop then
came before the grandchild existed, and the test failed in a full run but passed alone. A
negative control (taskkill without
/T) still fails it.
Changed
verify_runnerrefuses a cap set twice for one lane (max_flashwithlane_max.flash, ormax_opuswithlane_max.opus). verify applieslane_maxover the other two, so one value was dropped.- The README is written for a user of the server. It explains what verify does, gives a quick start
in Claude Code (a verify checkout, its pi runtime, the plugin, a first task), describes the task
workspace and each output file, and adds workflows, a defaults table, a Windows section, a
security section and a source map. It now states that the driver needs pi
(
harness/scripts/setup_pi.sh), thatenv: "none"is required off Linux, and that an MCP host can start the server with a reduced environment. - Every exported symbol in
src/has a doc comment:repo-tools docs check srcpasses with no MUST or SHOULD issue. It reported 63 symbols with no comment. - The README gives the output tail as 65536 characters. It said 64 KiB, but the tail is cut from the decoded text, not from the bytes.
- CI runs the tests on
windows-latestas well asubuntu-latest. The Windows process-tree kill and the Windows path handling had no CI coverage. - Dependabot proposes weekly bumps for the SHA-pinned GitHub Actions. Dependabot alerts and CodeQL default setup are on for the repository.
Security
- The verify pin takes verify's symlink hardening. The verifier can write
out/in its task workspace. verify now refuses a symlink there: a grader does not read through one (verify #15), and the driver does not delete, write, create or list through one on the host (verify #16). What the verifier leaves underout/no longer stops a task before its result is recorded (verify #16). The pin also takes the review fixes of verify #12 and #13, among them an SB2 deliverable that is a symlink is refused, and the jail stops when a read-only remount fails.
[0.1.1] - 2026-10-04
Security
verify_runnerrejects arun_nameor a pool incellsthat is not one path segment. verify joins both underVERIHARNESS_RUNSand deletes an existing task workspace there, so a value such as../xorwb:..made the runner delete directories outside the runs directory. A segment is letters, digits,.,_and-, not starting with a dot.- A timeout or a client cancel kills the whole process tree. verify's driver starts each agent
turn as a detached child, which leads its own process group on POSIX and leaves the parent's
job object on Windows, so the old group kill left the turn running. POSIX now takes a
pssnapshot of the descendants before it signals them; Windows usestaskkill /T /F.
Fixed
-
verify_runnerrejects alaneother thanflashoropusand acell_capentry whose cap is below 1 or whose key is not a bench ordefault. verify accepted both, and its scheduler then never started the cell's tasks and waited forever. -
The
skip_inflightdescription gives the unit as minutes. It said seconds. -
verify_read_resultreads at mostmax_bytesplus one byte of a text artifact. It read the whole file to return the prefix, so a largedriver.logblocked the server and could exhaust its memory. -
verify_driver,verify_runner,verify_scoreandverify_materializecarrydestructiveHint: true. They overwrite or delete files, andfalsedeclares additive changes only. -
A call that times out and is then cancelled is stopped once. The second stop armed a
SIGKILLtimer that was never cleared and fired at a stale list of process IDs. -
The Claude Code plugin installs and starts from a marketplace. It was the repository root, and
.mcp.jsonlaunchedbunonsrc/index.ts. A plugin cache clone has nonode_modules, so the server could not resolve@modelcontextprotocol/serverorzod, and the CLI lookup innode_modules/veriharnessfailed too. The plugin is nowplugin/: its.mcp.jsonlaunches a committed bundle,plugin/bundle/index.mjs, and setsVERIHARNESS_BINto a verify checkout at~/Github/verify. The marketplace entry'ssourceis"./plugin". -
tests/resolve.test.tspasses on Windows. Two assertions matched/path separators.
Added
bun run bundle(scripts/bundle.ts) buildsplugin/: the Bun-target bundle plus copies of the skill and the command. It builds from the repository root, so the output is byte-identical wherever it runs.tests/bundle.test.ts:plugin/must match a fresh build, and the bundle must start and list every tool from a directory with nonode_modulesin reach.tests/manifests.test.ts: every manifest must carry thepackage.jsonversion.
Removed
- The root
.mcp.json. Its only role was the Claude Code plugin's MCP registration, whichplugin/.mcp.jsonnow holds.
[0.1.0] - 2026-10-04
Added
- MCP server for the veriharness CLI on Bun, protocol revision
2026-07-28, with plugin manifests for Claude Code, Codex and Cursor.