ModelRefs / Batch Size — AI Glossary
Batch Size — AI Glossary
The number of independent inference requests processed simultaneously in a single GPU forward pass.
Overview
Larger batch sizes improve GPU utilization and throughput (tokens/s) but increase memory pressure and per-request latency. Continuous batching dynamically adds new requests to in-flight batches at the token level, improving GPU utilization for heterogeneous request lengths. Optimal batch size depends on model size, available VRAM, and latency SLO.
Reference details
| Topic | inference |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Batch Size — AI Glossary.
Frequently asked questions
What is Batch Size?
The number of independent inference requests processed simultaneously in a single GPU forward pass.
What concepts are related to Batch Size?
Closely related concepts include throughput, chunked prefill, gpu memory.