ModelRefs / Pipeline Parallelism — AI Glossary
Pipeline Parallelism — AI Glossary
A distributed inference/training strategy partitioning model layers across GPUs, with each GPU processing a pipeline stage. Also called PP or pipeline parallel.
Overview
Pipeline parallelism (Megatron-LM, DeepSpeed) assigns consecutive transformer layers to different GPUs. Micro-batches flow through the pipeline; each GPU processes its layers before passing activations to the next. Reduces per-GPU memory by 1/N but introduces pipeline bubble overhead. Combined with tensor parallelism for large-scale training.
Reference details
| Topic | inference |
|---|---|
| Also known as | PP, pipeline parallel |
| Last reviewed | 2026-06-24 |
Related terms
Commonly confused with
Pipeline parallelism splits by layer, tensor parallelism splits inside each layer. That determines the network bill: pipeline passes activations point-to-point between stages and tolerates slower links, while tensor parallelism synchronises within every layer and effectively demands a fast intra-node interconnect. Pipeline's cost is idle stages waiting for work, which micro-batching reduces but does not remove.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pipeline Parallelism — AI Glossary.
Frequently asked questions
What is Pipeline Parallelism?
A distributed inference/training strategy partitioning model layers across GPUs, with each GPU processing a pipeline stage.
Is Pipeline Parallelism the same as PP?
Yes — PP, pipeline parallel are common aliases for Pipeline Parallelism.
What concepts are related to Pipeline Parallelism?
Closely related concepts include model parallelism, distributed inference, disaggregated serving.