ModelRefs / ZeRO Optimizer — AI Glossary
ZeRO Optimizer — AI Glossary
A memory-optimization strategy partitioning optimizer states, gradients, and parameters across data-parallel workers to train huge models.
Overview
ZeRO (Zero Redundancy Optimizer, Rajbhandari et al. 2020) has three stages: Stage 1 shards optimizer states; Stage 2 adds gradient sharding; Stage 3 shards parameters. ZeRO-3 reduces per-GPU memory by N× (N workers), theoretically eliminating the memory barrier for model size. Implemented in DeepSpeed and approximated by PyTorch FSDP.
Reference details
| Topic | training |
|---|---|
| Also known as | ZeRO, ZeRO-3 |
| Last reviewed | 2026-06-24 |
Related terms
Example: Stage one removes three quarters of it
Mixed-precision training with Adam holds about 16 bytes per parameter: 2 for the fp16 weights, 2 for the gradients, and 12 for the fp32 master copy plus the two optimizer moments. The optimizer states alone are 12 of those 16 — 75% of the per-parameter memory. Stage 1 shards exactly that, so it delivers the largest single reduction and costs the least communication. Stage 2 adds gradient sharding, stage 3 adds the parameters and buys the most memory at the highest network cost. Work down the stages: take stage 1, measure, and only escalate when the model still does not fit.
Commonly confused with
ZeRO is the technique; DeepSpeed is the library that introduced it and PyTorch FSDP is a native implementation of the same idea. It is also not model parallelism: the computation is not distributed, only the storage — full layers are reconstructed on demand and then discarded, which is why the communication pattern is all-gather rather than the per-layer synchronisation tensor parallelism needs.
When to use it
Reach for it when:
- Training where optimizer state, not the weights themselves, is what exceeds memory
- Starting at stage 1 and escalating only as far as the model actually requires
- Fast interconnect, where the gather traffic of the higher stages is affordable
Reach for something else when:
- Parameter-efficient fine-tuning, which removes the optimizer-state problem outright
- Jumping to stage 3 by default — you pay maximum communication for memory you may not need
- Inference, which has entirely different memory characteristics
Primary source
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to ZeRO Optimizer — AI Glossary.
Frequently asked questions
What is ZeRO Optimizer?
A memory-optimization strategy partitioning optimizer states, gradients, and parameters across data-parallel workers to train huge models.
Is ZeRO Optimizer the same as ZeRO?
Yes — ZeRO, ZeRO-3 are common aliases for ZeRO Optimizer.
What concepts are related to ZeRO Optimizer?
Closely related concepts include deepspeed, fsdp, gradient checkpointing.