ModelRefs / Memory Bandwidth — AI Glossary
Memory Bandwidth — AI Glossary
The rate at which a GPU can transfer data between HBM and compute cores; the binding constraint for LLM decode throughput. 3 TB/s. Also called HBM bandwidth.
Overview
Decode is memory-bandwidth-bound because each generated token requires reading all model weights (hundreds of GBs) from HBM. H100 SXM offers 3.35 TB/s; A100 is 2.0 TB/s; MI300X is 5.3 TB/s. Techniques improving arithmetic intensity per byte (speculative decoding, batching, quantization) effectively increase usable bandwidth.
Reference details
| Topic | inference |
|---|---|
| Also known as | HBM bandwidth |
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Memory Bandwidth — AI Glossary.
Frequently asked questions
What is Memory Bandwidth?
The rate at which a GPU can transfer data between HBM and compute cores; the binding constraint for LLM decode throughput.
Is Memory Bandwidth the same as HBM bandwidth?
Yes — HBM bandwidth are common aliases for Memory Bandwidth.
What concepts are related to Memory Bandwidth?
Closely related concepts include gpu memory, tokens per second, quantization.