ModelRefs / Evaluation Harness — AI Glossary

Evaluation Harness — AI Glossary

A standardized framework for running reproducible LLM evaluations across benchmarks with consistent prompting and scoring.

Overview

EleutherAI's LM Evaluation Harness (lm-eval) is the reference: supports 200+ benchmarks (MMLU, ARC, HellaSwag, GSM8K, HumanEval) with standardized few-shot prompting, multiple-choice normalization, and generation-based scoring. Used by HuggingFace Open LLM Leaderboard. Ensures apples-to-apples comparison across model checkpoints.

Reference details

Topic	ecosystem
Last reviewed	2026-06-24

Example: Why the same benchmark gives different numbers

Two teams report MMLU for the same checkpoint and differ by several points. Neither is lying: few-shot count, prompt template, answer-extraction rule and normalisation all differ. A harness fixes those choices so the comparison is about the model. Without one, benchmark numbers are not comparable across sources.

Commonly confused with

The harness is the runner, not the benchmark. The benchmark is the dataset and scoring rule; the harness standardises prompting, extraction and normalisation around it. Reporting a benchmark score without naming the harness and settings omits most of what makes it reproducible.

When to use it

Reach for it when:

Comparing checkpoints, quantisations or fine-tunes of your own
Publishing numbers others should be able to reproduce
Tracking a model over time under identical conditions

Reach for something else when:

As a substitute for evaluation on your own task and data
Comparing your harness run against a vendor's differently configured one

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Evaluation Harness — AI Glossary.

Frequently asked questions

What is Evaluation Harness?