ModelRefs / Data Parallelism — AI Glossary
Data Parallelism — AI Glossary
A distributed training strategy replicating the full model on each device and splitting the data batch across devices.
Overview
Data parallelism (DDP in PyTorch) runs identical model replicas on each GPU. Each device processes a mini-batch shard; gradients are all-reduced across devices after each step. Simple to implement and scales efficiently for models that fit on a single GPU. Combine with model parallelism when the model is too large for one device.
Reference details
| Topic | training |
|---|---|
| Also known as | DDP, distributed data parallel |
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Data Parallelism — AI Glossary.
Frequently asked questions
What is Data Parallelism?
A distributed training strategy replicating the full model on each device and splitting the data batch across devices.
Is Data Parallelism the same as DDP?
Yes — DDP, distributed data parallel are common aliases for Data Parallelism.
What concepts are related to Data Parallelism?
Closely related concepts include fsdp, model parallelism, pipeline parallelism.