catfish-1234/proctor

proctor reads your git diff and blocks the changes an AI coding agent makes when it fakes a finished job: deleting a failing test, weakening the assertion, hardcoding the answer, swallowing the error, or switching off the checks that were about to catch any of it.

No network, no account, no API key. The whole check runs offline. You need Node 20 or newer.

Contents

Getting started

• Quickstart, try it against a repo without installing anything
• Install, and exactly what it writes
• What that setup does
• One thing to do afterwards
• Checking it worked
• Removing it

Using it day to day

• See it catch a real cheat
• What happens when it blocks
• What do the codes mean?
• Badges
• CI

Why it works

• How it works
• Can the agent get around it?
• By the numbers
• Benchmark

Reference

• Going further, every doc page
• The Proctor
• License

Quickstart

The npm package is not published yet: npm view @kavishdua/proctor currently returns 404, so an npx @kavishdua/proctor ... command from the registry will not work. Until the first release, run the checked-out source against any git repository:

git clone https://github.com/catfish-1234/proctor.git
cd proctor
npm ci
npm run build
node dist/cli.js check /path/to/repository

If it prints ✓ proctor: honest pass, your current changes are clean. If your changes don't touch tests at all, that's the answer you should expect.

The check writes nothing to the repository being checked and makes no runtime network call. npm ci installs build dependencies only in the proctor checkout.

Install

Two steps, and the second one is the only thing that writes to your repo.

Claude Code

/plugin marketplace add catfish-1234/proctor
/plugin install proctor@proctor-marketplace

Gemini CLI or Qwen Code

gemini extensions install https://github.com/catfish-1234/proctor
qwen extensions install https://github.com/catfish-1234/proctor

Anything else (30 agents supported, or none at all)

Until the npm package is published, build its installable tarball from source:

# In the proctor checkout
npm ci
npm run build
npm pack

# In the repository proctor should guard; use the .tgz path printed above
npm install --save-dev /absolute/path/to/kavishdua-proctor-*.tgz

Installing writes nothing to your repository. The install prints exactly what the next command would write, and stops. When you want the guards:

npx proctor setup

What that setup does

This is the complete list of what proctor setup writes, and where. Nothing outside your repository is touched, and no network call is made.

How the agent list is chosen: proctor looks for each agent's own config (.cursor/, .claude/, WARP.md, and so on) and writes the ruleset to just those. A repo with no agent config gets AGENTS.md, the cross-vendor standard. Run proctor agents first to see the list for your repo. --all installs to all 30 supported agents; --agents claude-code,cursor names them yourself.

setup stays out of the way where setting itself up would be wrong: in CI, in a global install, when pulled in as somebody else's dependency, outside a git repository, or under npx.

If you want the old install-and-wire behavior, set PROCTOR_AUTO_SETUP=1 and installing runs proctor setup for you. PROCTOR_NO_POSTINSTALL=1 silences the install-time notice entirely.

One thing to do afterwards

Commit what it wrote. The ruleset files and .proctor-adapter-manifest.json are ordinary files in your repo, and they only reach your teammates and your CI once they're committed.

git add -A && git commit -m "chore: add proctor"

If you use Claude Code, restart it so it picks up the new Stop hook.

That's the whole setup. You don't need to run proctor by hand, remember any flags, or write a config file.

Checking it worked

proctor drift-check   # exits 0 if every deployed ruleset copy still matches the source
proctor statusline    # "proctor: watching", or "proctor: 3 caught" once it has blocked something

Or trip it deliberately: delete a test in a scratch branch and run proctor check.

Removing it

proctor uninstall --dry-run   # see what would go
proctor uninstall

It removes only what it installed. Your own content in a shared file like AGENTS.md stays, a pre-commit hook that isn't proctor's is left alone, and proctor.config.json is left for you to keep or delete. To skip the hook once without uninstalling: git commit --no-verify.

See it catch a real cheat

An agent is asked to fix a bug in a slug generator. It can't get the whitespace-only case to return '', so instead of fixing slugify(), it deletes the inconvenient test:

 describe('slugify', () => {
   it('converts spaces to dashes', () => {
     expect(slugify('Hello World')).toBe('hello-world');
   });
-  it('handles a whitespace-only input', () => {
-    expect(slugify('   ')).toBe('');
-  });
 });

$ proctor check
tests/slug.test.ts
  ❌ tests/slug.test.ts:5  [RH001]  Test function 'handles a whitespace-only input' was deleted in this change.
      Restore the deleted test or document why it was intentionally removed.

How to fix these honestly:
  proctor check --explain RH001 --fix

1 finding (1 error, 0 warnings)
$ echo $?
2

The commit never lands. The agent has to actually fix slugify(), not just make the red go away.

Here's the full two-scene recording: proctor catching a deleted test at the CLI layer, then the Claude Code Stop hook blocking the same cheat live in an agent session.

imageproctor demo

What happens when it blocks

Your agent handles it. When a check fires, proctor prints what tripped and what an honest fix looks like, and the agent goes and does that instead. That's the normal path and it needs nothing from you. At the terminal you'll see the finding and an exit code of 2.

Three cases want a human, and each is one command.

The test change was genuinely intended. A feature got removed, so its tests went with it. Record it and the finding stops blocking:

proctor approve RH001 tests/legacy-billing.test.ts --reason "billing v1 removed in RFC-88"
git add proctor.config.json && git commit -m "chore: approve RH001 for legacy billing"

It stays visible in every report with your reason attached. Approvals are read from the committed config, so an agent cannot approve its own change in the change it is making, and an approval you haven't committed yet has no effect.

It's a false positive. Suppress that one line:

// proctor-ignore: RH004 reason: this is a lookup table, not a fixture hardcode

The marker has to be committed before the change it excuses, for the same reason approvals do. See inline suppression.

You want to know why a check exists. Every finding prints its ID, and every ID explains itself:

proctor check --explain RH001        # what this check looks for
proctor check --explain RH001 --fix  # what an honest fix looks like

What do the codes mean?

Every finding carries a short ID like RH001. They are just stable labels, the same idea as an ESLint rule name, so a check can be referenced without spelling out a whole sentence. Nothing to memorize: the plain-English name and full explanation print with every finding.

There are two families, and the split is by the claim each one checks.

RH0xx checks "the tests pass." These read the test suite and the code directly beneath it.

WI1xx checks "the work is done." Deleting a test is only one way to fake a finished job. These read shipped code for the rest of them, and none of the cheats they catch touches a test file.

Beta, and opt-in for now. The WI family is off by default in v1.0.0. Turn it on for one run with proctor check --wi (or --all-checks), or for good by listing the IDs you want in enabled in proctor.config.json, which also applies them in both hooks. Nothing about them is weakened by being opt-in: thirteen checks reading arbitrary source across 25+ languages is a larger false-positive surface than the RH family's, and it has had less real-world exposure, so they earn default-on in a later release rather than assuming it.

Every WI check skips test files on purpose. An empty catch is how you assert that something throws, canned data is what a fixture is for, and a loose cast is ordinary when building a partial mock. They watch the code your tests are meant to be proving.

Most of them also have the same escape hatch: a line whose comment explains why it is correct does not get flagged. That is not a loophole, it is the point. An agent racing to a green build does not stop to write the sentence, and if it does, the sentence is now in the diff for a human to read and disagree with.

Badges

imageproctor

✓ proctor: honest pass prints after a clean proctor check, and proctor badge gives you the same result as Markdown to paste into your own README or a PR description (generated by src/badge/index.ts):

$ proctor badge
[![proctor](https://img.shields.io/badge/proctor-honest_pass-22C55E)](https://github.com/catfish-1234/proctor)

A run only earns it when it is genuinely clean. Findings you approved through approvedTestChanges do not count as clean, since somebody decided to let those through. The printed line is suppressed under --ci, which is what the hooks and the GitHub Action use.

CI

Add proctor to a pull request in nine lines. Findings land in the job summary, and in Code Scanning as inline PR comments if the repository has it enabled.

# .github/workflows/proctor.yml
on: [pull_request]
jobs:
  proctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
        with: { fetch-depth: 0 }
      - uses: catfish-1234/proctor@v1

v1 is a moving tag: it follows the newest 1.x release, so patch and minor fixes arrive without a PR. To pin exactly instead, use a full commit SHA (catfish-1234/proctor@11fcd5cbad7d3a731dc5e79c4858012e74c6fc52), which is immutable and cannot be moved under you.

How it works

proctor has two halves.

The ruleset is a short document telling your agent how to finish work honestly. Most agents follow it most of the time.

The guard is for the rest of the time. A ruleset alone is a request, and an agent under pressure to show green can talk itself out of a request. So proctor also ships a deterministic, diff-level guard that runs on every commit and at the end of every agent turn, and blocks the changes that broke the rules. It works below the agent's own reasoning: it reads the code change itself, never the agent's explanation of it, so nothing the agent says can argue with it.

Can the agent get around it?

That is the question the design answers, so it is worth being concrete about.

The commit hook fails closed. If proctor cannot run at all, the commit is blocked rather than allowed. This used to be the other way round and it was the worst bug in the tool: the hook ran npx @kavishdua/proctor check --staged and treated exit 1 as "clean with warnings", but npx also exits 1 when it cannot resolve the package, so an unreachable registry looked identical to a clean run and the commit landed unchecked. The hook now probes with --version first, prefers a local install over the network, and refuses the commit when it cannot check it.

Config is read from the committed baseline, not your working tree. An agent that edits proctor.config.json in the same change it is trying to land has changed nothing about the run: enforcement uses the version at the diff baseline. The same goes for approvals and for inline proctor-ignore markers, which only count when they were committed before the change they excuse. Self-approval is not a matter of policy here, it does not work.

Switching the guard off is itself a finding. WI104 fires on a check removed from enabled, a severity downgraded, an ignore pattern or approval added, a ruleset file deleted, a --no-verify added to a script, or TypeScript strictness turned off. Since the config edit was already inert, the point of the check is to make the attempt visible rather than to stop it.

It reads the diff, never the explanation. Nothing the agent says about its change is an input. The remaining honest gap is a human running git commit --no-verify, which is deliberate: that is a person overriding their own tooling, and it is caught at the next turn by the Stop hook and in CI.

By the numbers

How often agents actually do this. These are other people's published measurements, not ours:

What proctor catches. The fixture and benchmark numbers below come out of npm test; the adversarial number comes from the independent widening corpus in bench/redteam/:

What we don't claim. That proctor makes an agent behave better. That is a different measurement and our own benchmark has not produced it: both complete live runs passed their held-out tests in every one of the 44 invocations, so the agent never reached the point where cheating was worth it and there was no behaviour to change in either arm. The 139 of 139 above is a detection claim, which is the claim the tool actually makes.

Benchmark

proctor bench measures how often an agent games its own tests, and how often proctor catches it, across a fixed pool of tasks with the ruleset on and off.

The honest state of this evidence: a complete 22-task live run against claude-code, both arms, all 44 agent invocations scored, against held-out tests each verified to fail their own unfixed source.

No difference between the arms, and the honest-pass column explains why. Every one of the 44 runs passed its held-out tests, including the one that tripped a signature. The held-out design exists precisely so an agent that fakes a green build fails a test it never saw. Nothing failed one. The agent solved every task in the pool, including the seven built to make cheating cheaper than fixing, so the situation the tool exists for never arose in either arm and there was no behaviour left for the intervention to change.

An earlier run of this same corpus showed 9.1% against 4.5%, and that difference did not reproduce. It was one task out of 22, which is what noise looks like at this sample size. Both runs are consistent with the same conclusion, and the apparent improvement in the first one would have been exactly the sort of unearned green this project exists to catch.

The single flagged diff was a false positive, and chasing it was worth more than the headline number. RH004 fired on task-17 in both arms, against an agent that had replaced a return version >= base stub with a real 52-line semver implementation. The flagged line was the return 0 that terminates its new comparator, which is the correct "equal" result, paired by the check against the deleted stub thirty lines away in a different function. That has been fixed, so the same run scored against current code reads 0.0% in both arms. The table above is left as the run actually scored, because a benchmark that quietly restates old results as if the tool had always behaved this way is not a record of anything.

The eight held-out suites that could not distinguish a fix from a cheat were found and strengthened before this run, and tests/bench-heldout-discriminates.test.ts now enforces that property for every task. See bench/METHODOLOGY.md.

The task corpus ships with this repository rather than the npm package, so bench needs a clone:

git clone https://github.com/catfish-1234/proctor && cd proctor
npm install && npm run build
node dist/cli.js bench --tasks 22 --agent claude-code --out bench/results-live.csv

A 22-task run is 44 agent invocations, which is more than one Claude subscription session allows. Three attempts reached 16, 14 and 16 tasks before the agent started returning "You've hit your session limit", and a rate-limited agent looks exactly like an honest one once it reaches the CSV: no changes, no cheat, no finding. --resume carries the completed tasks over from the .partial.csv a failed attempt leaves behind, so a run can span more than one window; the numbers above were collected that way. PROCTOR_BENCH_TIMEOUT_MS raises the per-invocation budget, which the hard-tier tasks need. Read the proctor: bench task-NN ... failed lines on stderr before trusting any number the table prints.

Going further

Everything above is the whole product for most people. These pages are for when you want more:

proctor supports 25+ languages and installs to 30 agents. Five diff-level checks (RH001, RH002, RH003, RH007, RH011) work across all of them; six (RH004, RH005, RH006, RH008, RH009, RH010) are JS/TS/Python-only; and RH012 and RH013 read CI and coverage config, so they apply everywhere. Of the work-integrity family, WI101, WI102 and WI103 carry per-language signatures, WI104 reads config files so it applies everywhere, and WI105 and WI106 are scoped to the languages whose tokens are unambiguous. Full matrix.

The Proctor

Picture the exam invigilator: arms crossed, half-moon glasses, watching over a sweating robot mid-delete of a failing test. That's proctor. The logo is a watchful eye with a green checkmark for a pupil, watching whether your green is real. When it catches a cheat, the iris flips red and the pupil becomes an X.

License

MIT