decision-model-benchmark
Compare decision models (TypeSafe Jev, OpenAI Decisions, Microsoft-Decision-1, Gemini Decision, Clef, any System One compatible server) used as a security gate, and measure how easily a decision can be flipped. Decision models take a state plus typed questions with a closed answer set (yes/no, choice, score) and return probabilities, no text. One adapter speaks the three wire formats; every arm gets the same seeded labelled cases and adversarial variants (injection, padding, option reorder, misleading option rename, paraphrase, encoding, extra JSON fields). Reports fail-open and fail-closed rates, flip rate per attack axis beside the identical-re-run noise floor, targeted attack success, consistency, calibration, refusal and error rates, latency and input tokens, with 95% intervals and a paired comparison between arms. Use when asked whether a decision model can gate tool calls, screen for prompt injection, route alerts or enforce a content policy. For chat models use model-benchmark.
- License
- MIT
- Compatibility
- Python 3.10+, standard library only (no SDKs, no extra packages). Real runs need HTTPS access to each endpoint under test and its API key in an environment variable. The keyless smoke test needs neither.
Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/decision-model-benchmark/SKILL.md
- skills/decision-model-benchmark/adapter.py
- skills/decision-model-benchmark/corpus/policies.json
- skills/decision-model-benchmark/corpus/seed_cases.jsonl
- skills/decision-model-benchmark/lib/evalkit.py
- skills/decision-model-benchmark/metrics.py
- skills/decision-model-benchmark/mock_server.py
- skills/decision-model-benchmark/new_experiment.sh
- skills/decision-model-benchmark/references/bring-your-own-cases.md
- skills/decision-model-benchmark/references/external-suites.md
- skills/decision-model-benchmark/references/metrics.md
- skills/decision-model-benchmark/references/not-measured.md
- skills/decision-model-benchmark/references/wire-formats.md
- skills/decision-model-benchmark/report.py
- skills/decision-model-benchmark/run_bench.py
- skills/decision-model-benchmark/tests/conftest.py
- skills/decision-model-benchmark/tests/test_dmb_adapter.py
- skills/decision-model-benchmark/tests/test_dmb_metrics.py
- skills/decision-model-benchmark/tests/test_dmb_run_bench.py
- skills/decision-model-benchmark/tests/test_dmb_variants.py
- skills/decision-model-benchmark/variants.py
Every link opens the file at its source, pinned to the revision this page describes.