mad-science

imageskills.sh imagevalidate imageLicense: MIT

Hypothesis-driven data science skills for coding agents (Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI and more). They take a raw idea all the way to a validated, reviewed notebook and, when it earns it, to a paper.

mad-science is a set of 14 Agent Skills that teach a coding agent to run machine learning and data science experiments the way a careful scientist would: state a falsifiable hypothesis, write down the expected result before looking at the data, implement the experiment in a Jupyter notebook, and review it for reproducibility, data leakage and weak evidence before trusting the numbers.

Why

Coding agents are fast at writing notebooks and bad at doubting them. Left alone, they tune until a metric looks good and report it. mad-science adds the missing discipline:

• Hypothesis first. /hypothesys interviews you until the idea has premises, a baseline and a success metric you can fail.
• Expectations before results. /expectation-driven-science applies the TDD idea to analysis: predict, then verify.
• Small, tracked slices. /experiment-pipeline splits the work into experiments with status and dependencies, so nothing runs out of order.
• A review gate. /exp-review checks every notebook for reproducibility (fixed seeds, pinned dependencies), security, data leakage and code quality.
• Paper-ready output. /to-article turns the results into a structured scientific article.

What a session looks like

you   > /hypothesys Adding weather features will improve next-day demand forecasts.

agent > ❓ Q1 - Baseline: what does the current forecast use, and what is its
        error today?
        ➡️ Recommended: the production model's MAPE over the last 90 days.

        ❓ Q2 - Success metric: how much better counts as "improve"?
        ➡️ Recommended: at least 2 points lower MAPE on a time-based holdout.
        ...

When the interview ends, /to-doc writes the experiment contract, /experiment-pipeline slices it, and /let-it-rip implements each slice in a notebook that /exp-review must pass.

Installation

A plugin updates itself. skills.sh copies editable files into your project, and you update them by hand. Pick one per agent: installing both gives you every skill twice.

1. Get the skills

Claude Code

claude plugin marketplace add samuhs/mad-science
claude plugin install mad-science@samuhs

Or inside a session: /plugin marketplace add samuhs/mad-science and /plugin install mad-science@samuhs. Claude Code leaves auto-update off for third-party marketplaces. Update with claude plugin update mad-science@samuhs, or turn auto-update on under /plugin → Marketplaces.

Codex

codex plugin marketplace add samuhs/mad-science
codex plugin add mad-science@samuhs

GitHub Copilot (CLI and VS Code)

copilot plugin marketplace add samuhs/mad-science
copilot plugin install mad-science@samuhs

In VS Code: Chat: Install Plugin From Source → https://github.com/samuhs/mad-science.

Cursor

Load the repo as a local plugin, or use skills.sh (below) with -a cursor.

Gemini CLI (manual updates)

gemini skills install https://github.com/samuhs/mad-science.git --path skills

Any other agent, or editable files (skills.sh)

npx skills@latest add samuhs/mad-science            # interactive: pick agents and skills
npx skills@latest add samuhs/mad-science -a codex   # cursor, opencode, windsurf, amp, ...
npx skills@latest add samuhs/mad-science --skill hypothesys --skill to-doc

When the installer asks which skills to take, include mad-science-setup and what-i-do. To update, run npx skills@latest update.

2. Run /mad-science-setup

Once per repository. It creates the layout and documents (docs/, coding standards, .env.example) the other skills expect.

3. Done

Start with /hypothesys. When you don't know the next step, ask /what-i-do.

The flow

/hypothesys → /to-doc → /experiment-pipeline → /let-it-rip → /exp-review
    idea      contract      slices/rounds         notebook       gate

Each step produces an artifact the next one consumes. The hypothesis becomes a contract, the contract becomes slices, each slice becomes a notebook, and each notebook is reviewed before moving on. The discipline underneath is /expectation-driven-science: write the expectation before trusting the data, the way TDD does for code.

Skill catalogue

Source of truth: skills/what-i-do/SKILL.md. If this README disagrees with it, what-i-do wins.

Main flow: Hypothesis → Notebook

Scientific discipline / reference

Utilities / operations

Quick guide: which skill?

FAQ

What are Agent Skills? Folders with a SKILL.md file that a coding agent loads on demand. Each skill describes when to use it and how to do one job. Claude Code, Codex, GitHub Copilot, Cursor and Gemini CLI all read the same format.

Which agents does mad-science work with? Claude Code, Codex and GitHub Copilot through the plugin marketplace, Cursor as a plugin, Gemini CLI through gemini skills install, and any other agent supported by skills.sh (OpenCode, Windsurf, Amp, ...).

How do I make a coding agent run reproducible ML experiments? Install the skills, run /mad-science-setup once, then start with /hypothesys. The flow forces fixed seeds, pinned dependencies and a written expectation before each result, and /exp-review blocks notebooks that fail those checks.

Do I need Jupyter? The implementation skills (/let-it-rip, /developing-data-science-notebooks, /exp-review) target Jupyter notebooks. The planning skills (/hypothesys, /to-doc, /experiment-pipeline, /literature-research) work in any repo.

Plugin or skills.sh? The plugin updates itself; skills.sh copies editable files into your project. Pick one per agent, not both.

Special thanks

The mad-science flow did not start from scratch. It adapts the work of two people to data science, and their results are the foundation of everything here:

• Matt Pocock, with mattpocock/skills. The interview → doc → slices → implementation → review cycle, the idea of "grilling" to sharpen your thinking, and the very shape of this repository as a multi-agent plugin all come from there. /hypothesys, /to-doc, /experiment-pipeline, /exp-review and /literature-research are direct adaptations of his skills.
• Lauren Tan (a.k.a. potato), with pstack. The philosophy of going deep before going fast, writing less but higher-quality code, and running rigorous agent workflows you can parallelize with confidence shaped the expectation-driven discipline and the review gates of this flow.

Thank you both for sharing this work openly. 🙏

Layout

.claude-plugin/
├── plugin.json          # plugin manifest (Claude Code, Copilot, Codex)
└── marketplace.json     # "samuhs" marketplace listing the "mad-science" plugin
.cursor-plugin/
└── plugin.json          # plugin manifest for Cursor
plugin.json              # portable manifest (agent-plugins.org) for Codex
llms.txt                 # LLM-readable summary of the repo (llmstxt.org)
assets/
└── social-preview.*     # repository social preview image (1280×640)
skills/
└── <skill-name>/
    ├── SKILL.md         # skill definition (frontmatter + instructions)
    ├── references/      # supporting material (optional)
    └── scripts/         # executable support + tests (optional)
scripts/
└── validate_plugin.py   # validates skills, catalogue and manifests

Contributing (governance)

1. Every change goes in through a Pull Request describing the proposed change. Never commit directly to main. Explain what changed, why, and the impact on the flow (if any).

2. Branches: type/description in kebab-case, e.g. feat/new-skill, docs/update-readme, fix/typo-what-i-do.

3. Commits: Conventional Commits (feat:, fix:, docs:, chore:, refactor:, test:), subject ≤ 72 characters.

4. Before finishing the PR, update the catalogue in skills/what-i-do/SKILL.md:

   • new skills go into the full catalogue (and into this README);
   • if the skill changes the flow, what-i-do must reflect the new flow (Main Flow and Flow Checks sections);
   • the PR must spell out the flow change for the reviewer.

5. Validate before opening the PR:

   python3 scripts/validate_plugin.py
   

6. Release: bump version in all three manifests (.claude-plugin/plugin.json, .cursor-plugin/plugin.json, plugin.json). The validator fails if they diverge.

Local development

claude --plugin-dir .                 # load the plugin straight from the checkout
claude plugin validate .              # validate manifest + marketplace

License

MIT