evaluating-llms-harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
- Version
- 1.0.0
- License
- MIT
Pinned to revision a74b0dbe5fa8, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/evaluating-llms-harness/SKILL.md
- skills/evaluating-llms-harness/references/api-evaluation.md
- skills/evaluating-llms-harness/references/benchmark-guide.md
- skills/evaluating-llms-harness/references/common-issues.md
- skills/evaluating-llms-harness/references/custom-tasks.md
- skills/evaluating-llms-harness/references/distributed-eval.md
- skills/evaluating-llms-harness/references/hardware-requirements.md
Every link opens the file at its source, pinned to the revision this page describes.