ModelRefs / Reward Model — AI Glossary

Reward Model — AI Glossary

A model trained to predict human preference scores for LLM outputs, used as the optimization target in RLHF.

Overview

The reward model (RM) is trained on human-ranked response pairs and outputs a scalar score. Quality of the RM is the primary bottleneck in RLHF — a poorly calibrated RM causes reward hacking in the policy.

Reference details

Topic	training
Last reviewed	2026-06-24

Example: It learns differences, not a scale

Training data is pairs: a human saw two responses to one prompt and said A is better than B. The model learns to score A above B — it never sees an absolute rating, so only the *difference* between two scores for the same prompt carries meaning. A score of 2.4 on one prompt and 1.1 on another says nothing about which response was better; they are different scales. This is why reward models are used to rank candidates within a prompt and why comparing raw reward values across prompts, or reading one as a quality percentage, is a category error.

Commonly confused with

A reward model predicts what a human would prefer; it does not know what is correct. Those diverge in a specific direction — preference data rewards confident, fluent, well-formatted answers, so the reward model learns to like them whether or not they are right. An LLM-as-judge does a similar job at evaluation time; the reward model is the version wired into training, where the policy can optimise against its blind spots.

When to use it

Reach for it when:

Preference-based alignment, as the signal a policy optimises against
Best-of-n sampling at inference: generate several candidates, keep the top-ranked
Filtering synthetic training data before fine-tuning on it

Reach for something else when:

As a quality score to report — the numbers are not calibrated or comparable across prompts
Optimising hard without a leash: the policy will find the model's errors, not human preference
Where correctness is verifiable — a test suite or checker is a better reward than a learned proxy

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Reward Model — AI Glossary.

Frequently asked questions

What is Reward Model?