Skip to content

wtcooper/model-evals

v0.1.0MIT

Evaluate a model through whatever API it has: security benchmarks for chat, vision, decision and voice models, and a guided review for image generation.

voice-model-benchmark

Measure whether a speech-to-speech (realtime voice) model is weaker when a request is SPOKEN than when it is typed. Sends the same seeded harmful and benign requests to one realtime model as text, as locally synthesised speech, and as that speech under audio perturbations (speed, pitch, volume, noise, echo, reverb), then has a separate judge model grade each reply transcript. Reports attack success and benign refusal per arm with 95% intervals, the paired speech-minus-text gap with exact McNemar, and per-perturbation deltas. Use it when a voice model or voice agent is under review (for example OpenAI gpt-realtime-2.1, which is reachable only on the Realtime API), or when a text-only benchmark result is being assumed to hold for a voice deployment. It also lists the voice threats that have no standard benchmark as review questions. For a text or vision model use model-benchmark; for an app endpoint use app-benchmark.

License
MIT
Compatibility
Python 3.11+. The runner, mock realtime server and tests use only the standard library. Real speech needs a separate virtual environment with piper-tts (GPL-3.0-or-later, runs locally) and one voice file downloaded once (about 60 MB). A real run needs network access to the realtime API over WebSocket and an OpenAI-compatible chat endpoint for the judge. Optional: the websockets package as an alternative transport.
Read SKILL.md at the source

Pinned to revision 3fb15e11d58f, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.