wtcooper/app-evals
Evaluate an LLM application's endpoint: a fixed adversarial benchmark and adaptive multi-turn red teaming.
Fast static security benchmark for an LLM app: runs a bundled, license-clean adversarial corpus (prompt injection, harmful, data leakage, over-refusal + M2S multi-turn) through promptfoo eval and reports F1 / recall / FPR / ASR by technique. Use to benchmark defenses, compare models/versions, or A/B/C a param-based guardrail. For live multi-turn attacks use app-redteam; for a guardrail API alone use control-isolate.
Adaptive multi-turn red team of an LLM app via promptfoo's crescendo attack engine: genuine multi-turn attacks that read the target's real responses and escalate on refusals, seeded by customized application objectives (with an optional cross-domain cyber pack). Runs fully locally via generate-once → redteam eval. Use to actively find vulnerabilities or stress-test multi-turn robustness. For a fast reproducible benchmark use app-benchmark; for a guardrail in isolation use control-isolate.