ModelRefs / PPO (Proximal Policy Optimization) — AI Glossary

PPO (Proximal Policy Optimization) — AI Glossary

The reinforcement learning algorithm used in RLHF to update the LLM policy against a reward model signal while preventing large policy deviations.

Overview

PPO (Schulman et al. 2017) clips the policy gradient update ratio, preventing destabilizing large steps. In RLHF, the LLM is the policy, the reward model provides the signal, and a KL divergence penalty against the SFT model prevents reward hacking. Used in InstructGPT, ChatGPT, and most early RLHF-trained models. Being replaced by GRPO and DPO for LLMs.

Reference details

Topic	training
Last reviewed	2026-06-24

Example: Four models in memory is why the field moved on

A PPO run holds the policy being trained, a frozen reference copy for the KL penalty, the reward model, and a value model — four sets of weights resident at once, plus their optimizer states. Direct preference optimisation reformulates the objective so the reward and value models disappear, leaving two. That is the practical reason DPO displaced PPO for most alignment work: not that PPO stopped working, but that halving the memory and deleting the reward-model training stage removes most of the operational difficulty. The KL term is what the whole apparatus exists to enforce — a leash to the reference model, so the policy cannot drift somewhere the reward model scores highly and humans do not.

Commonly confused with

PPO is a general reinforcement-learning algorithm; RLHF is the pipeline that uses it with a learned reward model. Confusing them makes it sound as though abandoning PPO means abandoning preference alignment — DPO and GRPO are different optimisers for the same goal. GRPO in particular drops the value model while keeping the sampling-based approach, which is why reasoning training uses it.

When to use it

Reach for it when:

Online preference optimisation, where fresh samples are scored during training
Where a reward model already exists and generating new samples is affordable
Verifiable-reward settings, where the reward is a checker rather than a learned model

Reach for something else when:

Straightforward preference alignment from a fixed dataset — DPO is far simpler
Without a KL penalty tuned and monitored: unleashed optimisation finds reward-model errors
Limited memory, where four resident models is the binding constraint

Primary source

Schulman et al., Proximal Policy Optimization Algorithms (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to PPO (Proximal Policy Optimization) — AI Glossary.

Frequently asked questions

What is PPO (Proximal Policy Optimization)?