Skip to content

wtcooper/app-evals

v0.1.0MIT

Evaluate an LLM application's endpoint: a fixed adversarial benchmark and adaptive multi-turn red teaming.

app-benchmark

Fast static security benchmark for an LLM app: runs a bundled, license-clean adversarial corpus (prompt injection, harmful, data leakage, over-refusal + M2S multi-turn) through promptfoo eval and reports F1 / recall / FPR / ASR by technique. Use to benchmark defenses, compare models/versions, or A/B/C a param-based guardrail. For live multi-turn attacks use app-redteam; for a guardrail API alone use control-isolate.

License
MIT
Compatibility
Requires python>=3.11, node>=18.
Read SKILL.md at the source

Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.