wtcooper/model-evals
v0.1.0MIT
Evaluate a model through whatever API it has: security benchmarks for chat, vision, decision and voice models, and a guided review for image generation.
scripts
- decision-model-benchmark
- adapter.py
- lib/evalkit.py
- metrics.py
- mock_server.py
- new_experiment.sh
- report.py
- run_bench.py
- tests/conftest.py
- tests/test_dmb_adapter.py
41 more
- tests/test_dmb_metrics.py
- tests/test_dmb_run_bench.py
- tests/test_dmb_variants.py
- variants.py
- image-model-review
- lib/evalkit.py
- scripts/_evidence.py
- scripts/make_fixtures.py
- scripts/provenance_check.py
- scripts/record_probes.py
- scripts/record_review.py
- tests/test_image_model_review.py
- model-benchmark
- lib/evalkit.py
- lib/inspect_report.py
- lib/inspect_runner.py
- lib/inspect_suite.py
- new_experiment.sh
- report.py
- run_suite.py
- tests/test_model_benchmark.py
- voice-model-benchmark
- adapters/__init__.py
- adapters/base.py
- adapters/openai_realtime.py
- audio.py
- cases.py
- corpus/build_subset.py
- import_clips.py
- judge.py
- lib/evalkit.py
- lib/manifest.py
- miniws.py
- mock_realtime_server.py
- new_experiment.sh
- perturb.py
- report.py
- run_benchmark.py
- synth.py
- tests/test_audio_and_corpus.py
- tests/test_realtime_and_metrics.py
plugin.json
- name
- model-evals
- version
- 0.1.0
- description
- Evaluate a model through whatever API it has: security benchmarks for chat, vision, decision and voice models, and a guided review for image generation.
- author
- ai-security-evals contributors
- license
- MIT
plugin.json
{
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
"name": "model-evals",
"version": "0.1.0",
"description": "Evaluate a model through whatever API it has: security benchmarks for chat, vision, decision and voice models, and a guided review for image generation.",
"author": {
"name": "ai-security-evals contributors"
},
"homepage": "https://github.com/wtcooper/ai-security-evals",
"repository": "https://github.com/wtcooper/ai-security-evals",
"license": "MIT",
"keywords": [
"ai-security",
"evaluation",
"model",
"benchmark",
"inspect-ai"
]
}
What else this package ships
These files come with the package and this site does not publish them. They are listed so you know what is there before you install it.
- 44Python files
- 3shell scripts
- 5JSON files
- probe_outcomes.example.csv
- 3other files