Pi LLM-as-a-Verifier

A bounded, read-only Pi integration inspired by LLM-as-a-Verifier. It has two explicit modes:

Pi-hosted CLI routes remain discrete. HTTP mode is the only true-logprob mode and never downgrades when its endpoint is absent.

Setup

The discrete runner needs Pi, /root/.local/bin/agy, and the fixed routes described in agents/llm-verifier.md. The full mode needs an endpoint that implements chat completions and token logprobs:

export PI_VERIFIER_LLM_ENDPOINT=http://localhost:8000/v1
export PI_VERIFIER_LLM_API_KEY=...       # never printed or logged
export PI_VERIFIER_LLM_MODEL=my-verifier
python3 bin/pi-llm-verifier.py --mode llm-verifier --operation compare --input compare.json

The request always includes logprobs=true and top_logprobs=20. Endpoint, model, timeout, and JSON extra headers can also be supplied with --endpoint, --model, --http-timeout, and --headers. Missing configuration returns machine-readable BLOCKED.

Operations

Default discrete selection:

python3 bin/pi-llm-verifier.py --input candidates.json

Full-fidelity selection uses criteria mappings, strings, criterion objects, or a Markdown path/name:

{"task":"fix the bug","criteria":{"correctness":"The fix works","verification":"Raw tests support it"},"candidates":[{"id":"a","trajectory":"..."},{"id":"b","trajectory":"..."}],"seed":7,"pivots":2,"n_evaluations":2}

python3 bin/pi-llm-verifier.py --mode llm-verifier --operation select --input select.json --cache /tmp/verifier-cache.json
python3 bin/pi-llm-verifier.py --mode llm-verifier --operation compare --input compare.json
python3 bin/pi-llm-verifier.py --mode llm-verifier --operation track --input track.json
python3 bin/pi-llm-verifier.py --mode llm-verifier --operation progress --input track.json

compare returns both rewards, criterion/repeat evidence, source markers, usage, and cache information. select runs a seeded Hamiltonian ring, chooses pivots, scores only non-pivot/pivot and pivot/pivot pairs, and reports the exact count N + k(N-k) + k(k-1)/2. Repeated slots alternate A/B, cache keys are directed, and one prefix is warmed before bounded concurrency.

Evidence, images, and progress

Candidates may be strings or structured steps. Structured steps preserve narration separately from raw stdout/stderr and include action, command, files, and image markers. Images accept local paths, base64/data values, and PNG/JPEG/GIF/WebP bytes. URL fetching is rejected unless --allow-url-images (or PI_VERIFIER_ALLOW_URL_IMAGES=1) is explicitly enabled. Size, count, MIME, prompt, output, history, and concurrency bounds are enforced.

Progress uses the opposite scale: A means no progress and T means complete. track scores selected checkpoints of a finished trajectory. ProgressTracker and the llm_progress_tracker Pi tool score only the prefix currently seen, with bounded create, update, result, and drop lifecycle. Future steps are never sent during an update.

Pi tools

The extension registers llm_verify_candidates, llm_compare_candidates, llm_track_progress, and llm_progress_tracker. All reports are bounded JSON. BLOCKED is a valid verdict and must not be converted into a winner. The verifier never edits, stages, commits, pushes, merges, or deploys a reviewed repository.

Load the native integration explicitly in Pi with:

pi --extension /absolute/path/to/pi-llm-verifier/extensions/llm-as-a-verifier.ts \
   --skill /absolute/path/to/pi-llm-verifier/skills/pi-llm-verifier

Alternatively use the Pi package/install path:

pi install /absolute/path/to/pi-llm-verifier
# or, after the npm publication:
pi install npm:pi-llm-verifier

The npm package is declared by package.json with the pi-package keyword and loads the native extension plus the portable skill. Pi installs the package resources, but the verifier still needs Python 3 and the configured verifier routes described above. The extension invokes the bundled runner with private temporary JSON input. Set PI_LLM_VERIFIER_BIN or PI_LLM_VERIFIER_PYTHON only when using a deliberately pinned installation elsewhere.

Build and publish the Pi package

The package catalog at pi.dev/packages indexes npm packages tagged pi-package. Validate the exact npm payload before publishing:

python3 -m pytest -q tests/test_pi_package.py
npm pack --dry-run --json
npm publish --access public

After publication, install the released package in a clean Pi profile and verify that the extension and skill load before announcing the catalog entry:

pi install npm:pi-llm-verifier
pi list

Do not publish credentials, private test state, .pi-subagents, caches, or reviewed repository contents. The package includes the Python runner because Pi loads the extension from npm, but Python and the external verifier commands remain host prerequisites.

Agent Plugins and MCP package

This repository also has a portable Agent Plugins 1.0.0 package. The standard coordinates packaging and component discovery; it does not standardize the verifier's scoring semantics, evidence requirements, or fail-closed decisions. Current package layout is:

plugin.json                              # Agent Plugins 1.0.0 manifest
mcp.json                                 # stdio MCP declaration
bin/pi-llm-verifier.py                   # shared runner and scoring modes
bin/pi-llm-verifier-mcp.py               # bounded JSON-RPC/MCP adapter
bin/validate_plugin_package.py            # offline stdlib package check
skills/pi-llm-verifier/SKILL.md           # portable Agent Skills contract
agents/llm-verifier.md                    # Pi-only routing/agent role
extensions/llm-as-a-verifier.ts          # native Pi extension

An Agent Plugins/MCP host should install the repository as a package and use its mcp.json. For a local checkout, validate and run the same components with:

python3 bin/validate_plugin_package.py
# The MCP host resolves this command with cwd=${PLUGIN_ROOT}:
./bin/pi-llm-verifier-mcp.py

The MCP server is stdio-only: stdout contains JSON-RPC responses and notifications produce no response; diagnostics are confined to stderr. It exposes pi_verify_candidates, pi_compare_candidates, and pi_track_progress. The MCP tools reuse the existing runner and return only SELECTED, OK, INPUT_OK, or BLOCKED. llm-verifier requires an explicit logprob-capable endpoint and valid token-level evidence; it never downgrades to pi-discrete-judge.

Agent Plugins hosts should use the package manifests for the MCP component, while Pi users should continue to use explicit --extension/--skill flags or Pi install paths. There is no claim of automatic Agent Plugins discovery by Pi.

Credential boundary

Provider endpoints, API keys, models, and optional headers are configured in the host environment, never in plugin.json, mcp.json, skill files, or MCP tool arguments. The true-logprob runner accepts PI_VERIFIER_LLM_ENDPOINT, PI_VERIFIER_LLM_API_KEY, PI_VERIFIER_LLM_MODEL, and optional PI_VERIFIER_LLM_HEADERS. The discrete route commands use PI_VERIFIER_GLM_CMD, PI_VERIFIER_AGY_CMD, and PI_VERIFIER_LUNA_CMD only for deliberate host/test configuration. Do not paste credential values into prompts, trajectories, reports, logs, or tracked files.

Safety and privacy

Secrets are rejected before dispatch where possible and redacted in reports/audits. API keys and extra headers are never included in reports. Temporary input and audit files are private; audit output is redacted. HTTP responses are bounded and transport failures, malformed score evidence, unsupported images, and missing endpoints fail closed. Unit tests use mocks only; no real API calls are required.

Criteria files

Use criteria/TEMPLATE.md:

## Ground Truth Note
Prefer observed output.

## Criteria
### Correctness {#correctness}
Describe what high and low scores mean.

HTML comments are stripped, IDs are slugged and deterministically de-duplicated, and empty criteria fail clearly.

License, citation, contribution

This adapter is MIT licensed; see LICENSE. It is inspired by the upstream project and its A–T fine-grained reward, progress, and probabilistic pivot-tournament machinery; see the upstream repository for the original paper and implementation. Contributions should remain stdlib/dependency-light, add deterministic offline tests, preserve fail-closed behavior, and run python3 -m pytest -q tests, python3 -m py_compile bin/pi-llm-verifier.py bin/pi_llm_verifier_full.py, the Node smoke, and git diff --check.