ModelRefs / How LLMs Are Trained — Tutorial

How LLMs Are Trained — Tutorial

Pre-training, supervised fine-tuning, RLHF, and DPO — the full pipeline from raw text to a useful assistant

Overview

Level: Advanced. Estimated reading time: 35 minutes.

Phase 1: Pre-training on next-token prediction

Pre-training is the most compute-intensive phase. The model is trained on hundreds of billions to trillions of tokens of text — web pages, books, code, scientific papers — to predict the next token at each position.

Scale: GPT-3 was trained on ~300B tokens. Llama 3 on 15T tokens. The rule of thumb from the Chinchilla paper (Hoffmann et al., 2022): train on ~20 tokens per parameter. A 7B model should see ~140B tokens for optimal compute efficiency.

Hardware: training frontier models requires thousands of H100 GPUs running for weeks or months. Parallelism strategies: data parallelism (same model, different data batches on each GPU), tensor parallelism (split individual matrices across GPUs), pipeline parallelism (different layers on different GPUs).

After pre-training, the model is a next-token predictor. It can complete text but doesn't follow instructions, refuses nothing, and isn't helpful in the conversational sense.

Phase 2: Supervised Fine-Tuning (SFT)

SFT trains the pre-trained model on high-quality examples of (instruction, response) pairs. Thousands to hundreds of thousands of examples written or curated by humans: "Write a summary of this article." → [summary].

SFT teaches the model the format and style of helpful responses. It's relatively cheap (few hundred GPU-hours) because the pre-trained model already understands language — SFT just teaches it to use that understanding in a question-answer format.

After SFT, the model follows instructions much better. But it doesn't yet have the nuanced judgment needed to refuse harmful requests, add appropriate caveats, or balance helpfulness vs safety.

Phase 3: RLHF and DPO

RLHF (Reinforcement Learning from Human Feedback, InstructGPT 2022) aligns the model with human preferences:

Step 1: collect preference data — show humans two model responses to the same prompt, have them pick the better one. Step 2: train a reward model on these comparisons to predict human preference scores. Step 3: use PPO (Proximal Policy Optimization) to fine-tune the LLM to maximise the reward model's score, with a KL penalty to stay close to the SFT model.

RLHF is complex and unstable. DPO (Direct Preference Optimization, 2023) achieves similar results without a separate reward model: directly optimise the LLM on the preference pairs. Mathematically equivalent to RLHF but simpler, more stable, and increasingly the preferred approach.

RLAIF (RL from AI Feedback): use a strong LLM as the rater instead of humans. Claude's Constitutional AI uses a constitution-based AI rater to scale feedback cheaply.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to How LLMs Are Trained — Tutorial.