Skip to content

wtcooper/model-evals

v0.1.0MIT

Evaluate a model through whatever API it has: security benchmarks for chat, vision, decision and voice models, and a guided review for image generation.

scripts

  • decision-model-benchmark
  • adapter.py
  • lib/evalkit.py
  • metrics.py
  • mock_server.py
  • new_experiment.sh
  • report.py
  • run_bench.py
  • tests/conftest.py
  • tests/test_dmb_adapter.py
  • 41 more
    • tests/test_dmb_metrics.py
    • tests/test_dmb_run_bench.py
    • tests/test_dmb_variants.py
    • variants.py
    • image-model-review
    • lib/evalkit.py
    • scripts/_evidence.py
    • scripts/make_fixtures.py
    • scripts/provenance_check.py
    • scripts/record_probes.py
    • scripts/record_review.py
    • tests/test_image_model_review.py
    • model-benchmark
    • lib/evalkit.py
    • lib/inspect_report.py
    • lib/inspect_runner.py
    • lib/inspect_suite.py
    • new_experiment.sh
    • report.py
    • run_suite.py
    • tests/test_model_benchmark.py
    • voice-model-benchmark
    • adapters/__init__.py
    • adapters/base.py
    • adapters/openai_realtime.py
    • audio.py
    • cases.py
    • corpus/build_subset.py
    • import_clips.py
    • judge.py
    • lib/evalkit.py
    • lib/manifest.py
    • miniws.py
    • mock_realtime_server.py
    • new_experiment.sh
    • perturb.py
    • report.py
    • run_benchmark.py
    • synth.py
    • tests/test_audio_and_corpus.py
    • tests/test_realtime_and_metrics.py
plugin.json
name
model-evals
version
0.1.0
description
Evaluate a model through whatever API it has: security benchmarks for chat, vision, decision and voice models, and a guided review for image generation.
author
ai-security-evals contributors
license
MIT

What else this package ships

These files come with the package and this site does not publish them. They are listed so you know what is there before you install it.

  • 44Python files
  • 3shell scripts
  • 5JSON files
  • probe_outcomes.example.csv
  • 3other files
View on GitHub