app-benchmark
Fast static security benchmark for an LLM app: runs a bundled, license-clean adversarial corpus (prompt injection, harmful, data leakage, over-refusal + M2S multi-turn) through promptfoo eval and reports F1 / recall / FPR / ASR by technique. Use to benchmark defenses, compare models/versions, or A/B/C a param-based guardrail. For live multi-turn attacks use app-redteam; for a guardrail API alone use control-isolate.
- License
- MIT
- Compatibility
- Requires python>=3.11, node>=18.
Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/app-benchmark/SKILL.md
- skills/app-benchmark/corpus/benign_overrefusal.json
- skills/app-benchmark/corpus/data_leakage.json
- skills/app-benchmark/corpus/harmful.json
- skills/app-benchmark/corpus/injection.json
- skills/app-benchmark/env.example
- skills/app-benchmark/install_dependencies.sh
- skills/app-benchmark/lib/extract_transcript.js
- skills/app-benchmark/lib/status_policy.js
- skills/app-benchmark/lib/summarize.py
- skills/app-benchmark/lib/transform_response.js
- skills/app-benchmark/new_experiment.sh
- skills/app-benchmark/promptfooconfig.browser.yaml
- skills/app-benchmark/promptfooconfig.yaml
- skills/app-benchmark/tests/mock_server.py
- skills/app-benchmark/tests/test_app_benchmark.py
Every link opens the file at its source, pinned to the revision this page describes.