Hungry Crab

                              o    .
                           .   o   .   O
     /\                                                     /\
    /  \       _____________________________________       /  \
   |    |     /                                     \     |    |
   | /\ |     |     \\\\\\\             ///////     |     | /\ |
   | \/ |     |      (@@@)               (@@@)      |     | \/ |
    \  /      |                                     |      \  /
     \ \      |      /\/\/\/\/\/\/\/\/\/\/\/\/      |      / /
      \ \_____|                                     |_____/ /
       \______\_____________________________________/______/
                  /     /      |   |      \     \
                 /     /       |   |       \     \

Eat a foreign repository. Digest it. Serve what is worth keeping, legally.

imageCI imageRelease imageLicense: MIT imagePython 3.11+ imageConventional Commits

Somewhere out there is a repository that fixed your flaky CI two years ago, wrote the AGENTS.md you keep meaning to write, and learned the hard way which file breaks every single time anyone touches it. Reading it properly costs you an afternoon. Reading fifty of them costs you a month.

The crab reads them for you. It drags the prey into a local cache, dissects it with fifteen deterministic miners, and boils a whole repository down to a digest small enough for an agent to actually read. Then it holds that digest against your repository and serves a ranked menu: what they have, what you lack, what it would cost you, and exactly what their license lets you take. Approve a few and they land as issues in your tracker, each with evidence links and a trace footer. Say no to one and the crab remembers, so it never offers it again.

The prey is never executed. Not one line of its text reaches your issues unless the license says it may.

Status: 0.6.8 is the latest release, adding behavior-planned donor searches, runtime coverage, safer loop recovery and actionable MCP diagnostics on 0.6.0 Everywhere, a limited stable analysis release. The CLI is available on PyPI. Update the CLI and plugins for bounded Windows/Linux acquisition, authenticated notices, runtime evidence and source-linked behavior findings (see the changelog). Standalone catch supports an explicitly selected fixed-capacity volume; without one it remains unbounded. Agent-host isolation remains deferred under #243. The supported scope is deterministic, human-supervised analysis: the CLI, scoped local MCP, portable skills and offline HTML reports. PR publication, Scheduled Crab work autonomy and measured learning/usefulness remain experimental; their live acceptance criteria are deferred to post-release work, not declared complete. Start with the Everywhere guide and the release record. The 0.6.5 execution report records the bounded public replay and its remaining unexplored facets.

The metaphor, in five words

Install

Prerequisites: Python 3.11+, git, and gh authenticated for the GitHub API.

The CLI, which every agent then calls, is the hungry-crab package on PyPI:

uv tool install hungry-crab

pipx install hungry-crab works as well. Either installs the latest release, and uv tool upgrade hungry-crab (or pipx upgrade hungry-crab) takes the next one.

The installation keeps PyYAML as its only runtime dependency. GitLab.com public reads do not require gh; private reads use GITLAB_TOKEN. GitHub Discussions and JUnit artifact reads need the corresponding authenticated API permissions.

To track master instead, which is always green and carries fixes before they are released, install from this repository:

uv tool install "hungry-crab @ git+https://github.com/drevendev/HungryCrab"

Claude Code, which adds the eat/license/serve/cleanroom/loop skills, sniff/menu commands, three subagents and two PreToolUse hooks:

claude plugin marketplace add drevendev/HungryCrab

claude plugin install crab@hungry-crab

Claude Code offers a plugin update when the plugin's version changes, which happens at each release rather than at every commit to master. It does not refresh third-party marketplaces on its own: turn that on once in /plugin → Marketplaces → hungry-crab → Enable auto-update, or refresh by hand with claude plugin marketplace update hungry-crab followed by claude plugin update crab@hungry-crab.

For development snapshots, run crab update to compare installed revisions as well as versions. A same-version snapshot can still be stale; the report gives a refresh/reinstall path and verifies the revision after applying it. Missing provenance and host hook activation are reported as UNKNOWN. Restart the agent and verify hook loading after a refresh.

Codex, same plugin, same marketplace:

codex plugin marketplace add drevendev/HungryCrab

codex plugin add crab@hungry-crab

Restart the agent session afterwards so it picks the plugin up. Cursor and anything else that reads the open SKILL.md format can use the skills/ folder of this repository directly; the only hard requirement is crab on PATH. The two safety hooks need no install of their own: each runs hooks/guard.py from the plugin with the first Python 3.11+ the shell finds (python, python3, or one uv python find knows), and the launcher imports the guard from the plugin's own src/, so a plugin-only install is guarded too.

To see what is out of date across the CLI and every agent you have, ask the crab:

crab update

It reports the installed and available versions (the latest release on PyPI for a PyPI install, master for one from git), notices which agents are on this machine and whether the plugin is installed in each, and prints exactly what to run. crab update --run does the plugin work for you.

Verify the CLI installation

After installing the CLI, verify that the crab command is available without GitHub authentication or accessing a remote repository:

crab --version
crab --help

Both commands should exit with status 0. crab --version prints crab followed by the installed version, while crab --help displays usage information, available commands, and options.

These read-only smoke checks verify CLI availability and argument parsing only. They do not validate repository fetching, digest generation, comparisons, or GitHub issue/PR publishing.

For the complete workflow, see Feed the crab below. The first crab sniff example accesses GitHub and requires network connectivity. Authentication requirements depend on the operation; use only the permissions necessary for the intended workflow.

Feed the crab

For repeated improvement, Scheduled Crab persists one phase per scheduler wake-up: crab loop init, next, record, pause/resume and measured cost history. The /crab:loop skill supplies judgement; the CLI bounds publication and waits for human merges. Work needs explicit maw-owner consent. Live rollout evidence is tracked separately from tests.

With an agent, one line does the whole protocol: judge the menu, ask you, create the issues.

/crab:eat pallets/click

By hand, the same protocol is six commands:

crab sniff pallets/click --maw .                  # license, size, activity: is it worth eating?

crab init                                          # .crab.yml: hunger, serve policy, ledger mode

crab compare pallets/click --maw . --issues 300   # digest both sides, diff, score -> gap.md, menu.md

crab menu pallets/click --top 15 --category ci     # the ranked menu of nutrients

crab serve pallets/click --maw . --ids crab:ci:ci.cache --as dry-run   # preview the issue

crab ledger mark crab:tooling:tooling.renovate rejected --reason "dependabot is enough"

crab compare catches and digests the prey the first time; digests are addressed by commit SHA, so eating the same commit twice is free. Giants take --since 2y (history newer than two years) or --shallow (default branch, tree only).

Feeder: a menu in CI

For broader goals, use supervised discovery: crab explore freezes comparison inputs, records source/maw hypotheses and registers non-trait behavior nutrients under existing licence, origin and hunger guards. A structural menu is an index; zero cards does not establish NO_GAP. Bounded local acquisition covers Linux volumes and native Windows VHD/Job Object prerequisites.

Run crab eat <local-repo> --deterministic --maw . --no-wiki to export a complete meal without a model, issue creation or ledger writes. Remote Feeder acquisition requires a dedicated quota volume and a shared byte allowance. The Feeder guide shows the composite action, reusable workflow, weekly schedule, artifact outputs and strict mode.

GitHub Marketplace launch roadmap is a proposal, with real cross-repository tests and evidence-first case studies still to come. This link is not an announcement that the Feeder is listed or that its examples were already run.

Local MCP and HTML reports

Install with the optional [mcp] extra and run crab mcp --root /absolute/workspace to expose crab_digest and crab_menu. Local roots are explicit, network reads are opt-in and neither tool publishes or writes the ledger. crab mcp --check checks the optional install and exits.

crab report /path/to/digest-or-meal --out report.html creates a self-contained browser report from existing artifacts, with no network, JavaScript or model calls. See the Everywhere guide for the development-candidate install command, Codex/Cursor/Claude configurations, portable skills and the clean-install verification.

Configuration

crab init writes .crab.yml in the maw. Every key is optional; omitted keys use the defaults below. Unknown top-level, hunger, budget, serve, and trust keys are usage errors (exit 2), with a suggested spelling where available; scoring retains its own flexible nested schema. serve.max_prs_per_run must be a non-negative integer. trust.same_owner and trust.bypass_license accept YAML booleans only: write false, not the string "false".

• license: SPDX id for the maw, or auto-detect from LICENSE when omitted.
• mode: normal or strict; strict downgrades COPY and COPY_FILE code to clean-room REIMPLEMENT while recognized configurations and templates stay copyable.
• profile: reviewed hunger defaults for balanced, library, CLI, service or frontend projects.
• memory: bounded learning from confirmed owner decisions, with an explicit off switch.
• hunt: bounded GitHub discovery queries, size/stars/license filters and revisiting policy.
• hunger: enable, disable, or cap each nutrient category with issues-only / ideas-only.
• ignore: maw-side git-style paths excluded from its own digest, in crab compare and in crab digest <maw> --maw <maw>; a prey's own .crab.yml is data and is never read.
• budget: whole-digest Markdown budget policy: warn (default), enforce, or off.
• serve: issue/PR serving policy, labels, assignees and token source; serve.prs gates crab serve --as pr-branch, which publishes REIMPLEMENT nutrients from clean-room receipts and COPY nutrients from materialization receipts, with their attribution.
• trust: same-owner and explicit-owner relationships plus the visible license-bypass escape hatch.
• attribution_file: the notice file for copied sources, rendered by crab attribution from the receipts in .crab/attributions.json that crab serve --as pr-branch writes when it carries prey files in; crab attribution --check is the CI gate. The file is headings and paragraphs, plus the source's licence and NOTICE texts in fenced blocks, so that a Markdown formatter's defaults leave it alone; if yours rewrites it anyway, exclude the file from the formatter, not from the check.
• ledger: store meal history in the repository, cache, or nowhere.
• scoring: per-section overrides for data/scoring.yml; crab tune can suggest them.
• loop: scheduled phase policy, owner work consent, fixed prey, budgets and recovery limits; see Scheduled Crab.

This is the current commented template written by crab init:

# Hungry Crab maw configuration. Every key is optional; these are the defaults.
license: null              # SPDX id of this repository; detected from LICENSE when null
mode: normal               # normal | strict. Strict downgrades COPY code to clean-room
                           # REIMPLEMENT; configs and templates remain copyable.
profile: balanced          # balanced | library | cli | service | frontend; explicit hunger wins
memory:
  enabled: true            # confirmed decisions influence ranking; explicit weights still win
  min_decisions: 3
  strength: 0.3            # bounded multiplier; never changes license or hunger ceilings
hunt:
  queries: []              # derive up to four GitHub searches from enabled maw gaps
  exclude: []
  licenses: []
  min_stars: 20
  max_repo_kb: 307200
  max_candidates: 50
  limit: 10
  include_seen: false
  allow_unknown_size: false
hunger:                    # per nutrient category: true | false | issues-only | ideas-only
  security: true
  ci: true
  tests: true
  tooling: true
  ai-config: true
  hygiene: true
  docs: true
  deps: true
  history-lesson: true
  issue-lesson: true
  architecture: issues-only
  code: ideas-only         # source-linked behavior findings from crab explore register
ignore: []                 # globs excluded from this repository's own digest, so that test
                           # fixtures and vendored trees are not mistaken for your code, e.g.
                           # [tests/fixtures/**, examples/**]. Patterns are case-sensitive on
                           # every platform, like the git paths they match.
budget:
  policy: warn             # warn | enforce | off for the whole digest Markdown budget
serve:
  issues: ask              # auto | ask | off
  prs: ask                 # auto | ask | off; gates --as pr-branch (REIMPLEMENT and COPY)
  max_prs_per_run: 3
  publication_allow: []    # exact content-bound finding/reason entries; commit before use
  labels: [hungry-crab]
  assignees: []
  token_env: ""            # environment variable holding the token to file issues as, e.g.
                           # CRAB_BOT_TOKEN with a GitHub App installation token. Empty means
                           # gh's own login: your issues carry your name, not the crab's.
trust:                     # a license is a promise to strangers; these are not strangers
  same_owner: true         # prey owned by the account that owns this repository is your own code
  owners: []               # other accounts whose repositories count as your own, e.g. [acme-inc]
  bypass_license: false    # last resort: treat every prey as COPY. Every card says so, and the
                           # verdict is flagged for human review, because this is not a finding
                           # about the license but a decision to stop asking.
attribution_file: THIRD_PARTY_NOTICES.md
                           # where COPY records its sources: rendered by `crab attribution`
                           # from the receipts in .crab/attributions.json, which
                           # `crab serve --as pr-branch` writes when it carries prey files in.
ledger: repo               # repo (.crab/ledger.json, committed) | cache | none
scoring: {}                # overrides for data/scoring.yml sections; `crab tune` suggests them
loop:
  cadence: daily           # cadence belongs to your scheduler; the CLI never starts one
  autonomy: read           # read | serve | work; merging always belongs to a human
  work_authorized: false   # the maw owner must explicitly authorize work, including tags
  discovery: false         # opt in to a lease-bound HUNT shortlist
  prey: []                 # fixed allowlist; empty requires discovery
  budget:
    phases_per_day: 4
    prey_per_round: 2
    open_issues_max: 10
    open_prs_max: 2

What the miners extract

crab digest owner/repo writes a token-budgeted digest/ folder that an agent reads progressively:

manifest.json is the entry point: every file with a token estimate, the miners that ran, a small summary and the suggested reading order. Markdown is paged at 3,500 tokens per page by default (history.md, history.2.md, and so on; nothing is dropped), and the whole digest has a policy for its 30,000-token reading budget — warn by default, enforce for a budgeted loop, off for a human, set under budget in .crab.yml; JSON keeps the full data for scripts.

For crab digest --out, use an empty directory or an existing digest directory. The crab replaces and removes only artifacts recorded in that directory's previous valid manifest. Other files are preserved; a conflicting caller-owned name is refused before publication, including case-insensitive collisions.

Guiding principle: scripts squeeze out everything that can be squeezed deterministically; the model is spent only on judgment.

From digest to issues

• compare turns the two digests into candidate nutrients: trait rules (ci, security, tooling, hygiene, docs, ai-config, tests), tool and dependency diffs within shared ecosystems, README sections, commit and changelog discipline, history and issue lessons, architecture raw material. Every candidate carries a stable id, evidence linked to the prey commit, effort, risk and a license mode.
• scoring is a formula you can read: category weight x value x uptake x license mode x effort, minus risk. Weights live in data/scoring.yml and are overridable per maw.
• serve creates issues with a hidden crab:<id> marker, a label and a trace footer, so a rerun creates no duplicates. The why and how come from the agent through a notes file.
• ledger (.crab/ledger.json by default) remembers every meal and decision; rejected and served nutrients vanish from later menus. crab tune reads it and tells you which weights to move, which categories to switch off, and which prey were a waste of time.

Licenses are decided by an engine, not an opinion

A deterministic maw license x prey license matrix gives every nutrient one mode:

The three cases where the engine will not help you look alike and are not:

Issue and discussion text is always IDEAS_ONLY, because the copyright belongs to the commenters. This is a compliance aid, not legal advice.

Safety

• Prey code is never executed. The miners read files and run read-only git plumbing. No npm install, pytest or make ever runs inside the cache.
• Prey content is untrusted data. Markdown summaries carry structure (headings, names, counts), never the body of README or agent-instruction files, and instruction-like fragments are flagged in the JSON.
• Least privilege. sniff and catch need read access to the GitHub API; serve uses your own authenticated gh and creates nothing until you ask it to.
• Two PreToolUse hooks make the first rule mechanical in Claude Code. crab-prey-guard refuses a Bash or PowerShell command that touches the prey cache unless it is one of a few audited read-only shapes (in PowerShell, a read cmdlet such as Get-Content or Get-ChildItem with literal arguments). It also checks Monitor and desktop terminal commands, and refuses Write, Edit, MultiEdit and NotebookEdit calls into the cache. The clean-room guard keeps the implementer out of the cache through Read, Grep, Glob, Write, Edit and PowerShell; the prey guard covers its Bash commands. They run from the plugin's own source tree through hooks/guard.py, with the first Python 3.11+ the shell finds, so nothing has to be installed or signed for them; when not even an interpreter is there, the hook says so and the call proceeds, because a guard that blocked every call over its own setup would be switched off rather than fixed. Codex does not run them at all: a plugin with a root plugin.json is loaded through its Agent Plugins loader, which has no hook support (openai/codex#39895). The prey guard has been observed refusing a command in a live session; the clean-room guard has not yet (#83, #141).

Benchmarks

uv run python benchmarks/run.py pallets/click colinhacks/zod

The milestone limits are 120 seconds and 30,000 Markdown tokens per digest. Results land in benchmarks/results/; see benchmarks/README.md.

Development

uv sync                      # environment with the dev group

uv run pytest                # tests build three synthetic git repositories as fixtures

uv run ruff check . && uv run ruff format --check . && uv run mypy

uv run crab digest . --maw . --out digest-out --maw-license MIT   # the crab eats itself

See CONTRIBUTING.md and AGENTS.md, the guide coding agents read.

Design

The design documents live in docs/design:

• Concept and skill form factor: the three layers, the digest format, the license engine, security, the decisions log.
• MVP: commands, miners, milestones 0.1 to 0.3, acceptance criteria.
• Roadmap: base crab, evolving crab, forks.
• Evolving Hungry Crab: the self-improving loop.

License

MIT. The tool helps with license compliance; it is not legal advice.