ModelRefs / Tensor Parallelism — AI Glossary
Tensor Parallelism — AI Glossary
A model-parallelism strategy that splits individual weight matrices across multiple GPUs, enabling inference of models too large for a single GPU.
Overview
Tensor parallelism (used by Megatron-LM, vLLM, TensorRT-LLM) shards each transformer layer across N GPUs. Each GPU processes a fraction of the matrix and all-reduces results. Best for latency-sensitive single-request scenarios.
Reference details
| Topic | inference |
|---|---|
| Also known as | model parallelism, TP |
| Last reviewed | 2026-06-24 |
Related terms
Commonly confused with
The counterpart to pipeline parallelism: it splits individual weight matrices rather than assigning whole layers, so every device works on every layer and they must synchronise inside each one. That gives lower latency than pipelining and much heavier communication, which is why it is normally kept within a single node. Both sit under model parallelism, which is the general term.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tensor Parallelism — AI Glossary.
Frequently asked questions
What is Tensor Parallelism?
A model-parallelism strategy that splits individual weight matrices across multiple GPUs, enabling inference of models too large for a single GPU.
Is Tensor Parallelism the same as model parallelism?
Yes — model parallelism, TP are common aliases for Tensor Parallelism.
What concepts are related to Tensor Parallelism?
Closely related concepts include inference, throughput, vllm.