heyman333/jev

Your agent makes narrow judgments all day long — is this file relevant? is this test failure a flake? which bucket does this go in? Every one of those is an LLM call, tokens, and seconds.

jev doesn't generate text. It returns a typed value with a calibrated confidence, in about $0.00004 and 300–500ms.

Three questions, one call, one charge.

Why not just ask the LLM

Need the auth-related files out of 184?

git ls-files | jev --batch --bool relevant "Is this file part of the auth logic?"

{"input":"src/auth/session.ts","relevant":true}
{"input":"README.md","relevant":false}

Your agent reads 12 files instead of 184 — and never had to guess.

Measured

Not estimates. Both paths run through the same gateway, so the billing numbers come straight out of the responses. The script and the inputs are in bench/ — re-run it yourself.

20 Korean support tickets → which team, and is it urgent.

9.3× cheaper, against the LLM's best case — one batched call, no per-item overhead. Per judgment jev came to $0.0000167 and ~500ms.

jev spends 17× more tokens and still costs a tenth as much: its input tokens are about 160× cheaper, and its output is free. Sending each item separately, with the full question spec each time, is the cheap thing to do here.

What this doesn't show

• The batched LLM is faster in wall-clock time. One call answered all 20 in ~2.1s. If your items all fit in one prompt and you don't mind parsing the result, speed is not the reason to reach for jev — cost, a fixed output shape, and calibrated confidence are.
• Free-tier rate limits hit both. Under sustained concurrency both jev and the chat models returned 429. The benchmark runs sequentially so limits don't contaminate the comparison.
• Neither side is ground truth. 18/20 agreement means they differed on two tickets, not that either was right.

AI_GATEWAY_API_KEY=... node bench/run.mjs openai/gpt-5.2 2

What it includes

Copy the skill directory anywhere and it works. No build step, no npm install — just Node 20+ and its built-in fetch.

Install

jev ships as a portable Agent Plugin — one plugin.json, one skills/ directory. Codex and Claude Code install the same files.

Get a key

Either one works. Both are plain HTTPS — neither adds a dependency.

If you already have a Vercel account the second one is faster; AI Gateway serves jev on the free tier. With both set, jev calls TypeSafe directly.

Put it in ~/.zshenv, not ~/.zshrc

This is where most people get stuck.

echo 'export AI_GATEWAY_API_KEY="vck_..."' >> ~/.zshenv

zsh reads each file under different conditions:

Agents run commands in a non-interactive shell. Put the key in ~/.zshrc and echo $AI_GATEWAY_API_KEY prints it in your terminal while the agent keeps reporting "no API key".

On bash, use ~/.bashrc, and add ~/.profile too if the same symptom shows up.

Restart the agent

A running process never sees an environment variable that changed after it started. The macOS desktop app does not quit when you close the window — ⌘Q, then reopen.

Verify

jev --doctor

키          ✓ AI_GATEWAY_API_KEY (60자)
백엔드      Vercel AI Gateway 경유
연결        ✓ 588ms

Codex

codex plugin marketplace add heyman333/jev-skill

Then pick jev in the plugin browser with /plugins. Or drop the directory in by hand:

mkdir -p ~/.agents/skills && cp -r skills/jev ~/.agents/skills/     # every project
mkdir -p .agents/skills && cp -r skills/jev .agents/skills/         # this repo only

Codex picks it up next session. Ask it to classify or filter something and it applies on its own; to force it, type $jev.

Claude Code

/plugin marketplace add heyman333/jev-skill
/plugin install jev@jev-skill

The default scope is user — all of your projects, nothing written to the repo. Narrow or widen it with --scope:

# just me, just this project (.claude/settings.local.json — auto-gitignored)
claude plugin install jev@jev-skill --scope local

# the whole team on this project (.claude/settings.json — commit it)
claude plugin install jev@jev-skill --scope project

Updating — refresh the marketplace first, then the plugin, then restart:

/plugin marketplace update jev-skill
/plugin update jev@jev-skill

Or copy the skill by hand into ~/.claude/skills/ or <project>/.claude/skills/.

Cursor, Gemini CLI, anything else

The skill is one Markdown file and one Node script with no agent-specific dependencies. Point your AGENTS.md or .cursor/rules at skills/jev/SKILL.md and it works.

Usage

Once installed, your agent reaches for it on its own.

> pick out the auth-related files in this repo
> of these failing tests, which are real bugs
> sort these issues into buckets

Or call it directly:

jev "<what to judge>" --bool  <id> "<question>"
jev "<what to judge>" --pick  <id> "<question>" <option>[=description] ...
jev "<what to judge>" --score <id> "<question>" <lowest> ... <highest>
cat items.txt | jev --batch --bool <id> "<question>"

A failing item in a batch doesn't take the rest down; that line comes back as {"error":...}.

As a shell gate:

if [ "$(jev "$(git log -1 --format=%B)" --quiet \
        --bool ok "Does this commit message explain what changed and why?")" = "false" ]; then
  echo "Rewrite the commit message"; exit 1
fi

Question wording decides accuracy

jev is never wrong. Ask a vague question and you don't get a vague answer — you get a confidently wrong one.

A real one from building this. Asked how hard the task "classify 100,000 reviews" was, jev answered 1.55/2. Classifying one review is trivial — it had read the volume as difficulty. Adding "judge ONE item; ignore how many there are" moved it to 0.21.

• Bad: "Is this task hard?"
• Good: "Is handling ONE of these items hard? Ignore the total count."

Describe your options. billing=charges, refunds, invoices beats billing.

Low confidence stops the line

--pick and --score return confidence. Under 60% the CLI says so.

team           billing  (71% · technical 14%, sales 14%)
               ⚠ team 확신도가 낮습니다. 사람이 보는 게 낫습니다.

The skill is told not to quietly take the top answer here — show the candidates or read that one item directly. An LLM sounds certain even when it isn't. jev hands you the number.

Troubleshooting

The key isn't picked up

jev --doctor

It finds which config file holds the key and tells you the cause and the fix. It never prints the key — only its length and whether it's there.

Don't check with echo $AI_GATEWAY_API_KEY. The value lands in your terminal and your logs. ${VAR:-none} is not masking either — if the variable is set, it prints the value.

It works in my terminal but not in the agent

The key is in ~/.zshrc. Agents use a non-interactive shell, which doesn't read that file. Move it to ~/.zshenv. If other ~/.zshrc variables (NVM_DIR, say) are also missing, that confirms it.

I updated the plugin but the old behavior persists

The plugin cache doesn't refresh on its own. Update the marketplace first, then the plugin, then restart the agent.

grep -c zshenv ~/.claude/plugins/cache/jev-skill/jev/*/skills/jev/jev.mjs

The low-confidence warning keeps firing

The question is ambiguous. Describe your options (billing=charges, refunds, invoices) and say "judge ONE item" for batch work. See Question wording decides accuracy.

Limits

• It can't make anything. Writing code or prose is still the LLM's job. jev only judges.
• It can't explain itself. No "why did you decide that" — just probabilities.
• It can't see past the text you hand it. Read the file first, then pass the content.
• One-offs aren't worth it. The round trip costs more than the judgment saves.

Repo layout

plugin.json                  # portable manifest (Agent Plugins 1.0) — the source of truth
.codex-plugin/               # Codex manifest
.claude-plugin/              # Claude Code manifest + marketplace catalog
.agents/plugins/             # Codex marketplace catalog
.agents/skills/jev -> ../../skills/jev    # a symlink, not a copy
skills/jev/                  # the skill — SKILL.md, jev.mjs
scripts/check-manifests.mjs  # six manifests, one name and one version
scripts/build-demo-svg.mjs   # docs/demo.svg is generated from real CLI output
bench/                       # the head-to-head benchmark behind the Measured section

Both runtimes load the same skills/. The skill is never forked per agent.

Development

npm test      # spawns a mock server and drives the CLI over real HTTP. No dependencies
npm run check # manifest name and version agreement

Bumping the version means editing five files. npm run check catches any drift.

License

MIT