ModelRefs / LLM-as-Judge — AI Glossary
LLM-as-Judge — AI Glossary
An evaluation method where a strong LLM scores another model's outputs against a rubric or criteria. A judge score is a model's opinion, not a measurement.
Overview
LLM-as-judge enables cheap pairwise comparisons and nuanced quality scoring at scale. Known biases: positional (prefers first option), verbosity (favors longer answers), and self-preference. Mitigations: swap order, calibrate against human labels, use multiple judges.
Reference details
| Topic | evaluation |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Example: The biases are measurable, so measure them
Present the same pair twice with the order swapped. If the judge picks whichever came first, you have positional bias and your scores are partly noise. The same test catches verbosity preference and self-preference. A judge you have not tested this way is an instrument of unknown accuracy.
Commonly confused with
A judge score is a model's opinion, not a measurement. It is reproducible enough to track direction over time, which is genuinely useful, but it is not ground truth and it moves when the judge model changes. Validate against human labels at least once before trusting it.
When to use it
Reach for it when:
- Quality is a judgement call rather than a checkable answer
- You need evaluation at a volume humans cannot cover
- Pairwise comparison is enough — it is more reliable than absolute scoring
Reach for something else when:
- Correctness is programmatically checkable; check it instead
- The judge has never been validated against human labels
- The judge is grading its own family's output, where self-preference is strongest
Referenced by
This term is used by the following ModelRefs references:
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LLM-as-Judge — AI Glossary.
Frequently asked questions
What is LLM-as-Judge?
An evaluation method where a strong LLM scores another model's outputs against a rubric or criteria.
What concepts are related to LLM-as-Judge?
Closely related concepts include eval, evaluation benchmark, g eval.