ModelRefs / LLM-as-Judge — Prompt Pattern
LLM-as-Judge — Prompt Pattern
Score model outputs with another LLM call using a rubric. Define a numeric rubric; use a strong model to judge candidate outputs. Backbone of automated evals.
Overview
Define a numeric rubric; use a strong model to judge candidate outputs. Backbone of automated evals.
When to use it: You need scalable scoring on open-ended tasks.
Pattern details
| Pattern class | evaluation |
|---|---|
| Difficulty | intermediate |
| Also known as | judge prompt, rubric judge |
| Last reviewed | 2026-06-07 |
Known failure modes
- Position bias — Pairwise judging favors the first option. Mitigation: Randomize order; average over both.
- Scale clustering — All scores cluster at 3-4. Mitigation: Use pairwise win rate instead of absolute scores.
When not to use it
- Using the same model as both generator and judge for the same task.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LLM-as-Judge — Prompt Pattern.
Frequently asked questions
When should I use the LLM-as-Judge prompt pattern?
You need scalable scoring on open-ended tasks.
What are common failure modes of LLM-as-Judge?
Position bias • Scale clustering
Does LLM-as-Judge work across models?
Yes when the anatomy is preserved — see model compatibility matrix on the pattern page.