ModelRefs / Preference Dataset — AI Glossary
Preference Dataset — AI Glossary
A dataset of paired model responses labeled with human preferences, used to train reward models for RLHF and DPO.
Overview
Preference datasets contain (prompt, chosen_response, rejected_response) triples. Human labelers rank model completions; the signal trains a reward model or is used directly in DPO/IPO. HH-RLHF (Anthropic), UltraFeedback, and Nectar are publicly available examples. Quality of preference labels is the bottleneck for alignment performance.
Reference details
| Topic | training |
|---|---|
| Also known as | RLHF dataset, comparison dataset |
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Preference Dataset — AI Glossary.
Frequently asked questions
What is Preference Dataset?
A dataset of paired model responses labeled with human preferences, used to train reward models for RLHF and DPO.
Is Preference Dataset the same as RLHF dataset?
Yes — RLHF dataset, comparison dataset are common aliases for Preference Dataset.
What concepts are related to Preference Dataset?
Closely related concepts include rlhf, dpo, instruction dataset.