experiment-arms
Runs many single-hypothesis "arms" as do-not-merge PRs, measures each with paired, interleaved runs against a cost-matched control, keeps only the arms that clear a keep bar set in advance, records every negative, and composes the winners into a stack that is re-measured on real traffic after merge. It opens PRs and burns CI, so it runs only on /lever arms or an explicit ask to measure ideas ("which of these ideas actually helps?", "is this change worth it?", "try that tip I saw", "run arms on this") for a CI, perf, eval, benchmark or cost hill-climb where the system can be run on demand.
Pinned to revision f859d0aa2be9, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/experiment-arms/SKILL.md
- skills/experiment-arms/ci-adapter.md
- skills/experiment-arms/scripts/arm_stats.py
- skills/experiment-arms/scripts/ci_arm_driver.py
- skills/experiment-arms/scripts/ci_arm_metrics.py
Every link opens the file at its source, pinned to the revision this page describes.