Throughput — AI Glossary

Reference details

Related terms

• Latency
• Inference
• Continuous Batching
• vLLM

Example: The trade you are actually making

Batch size 1 might give 30 tokens/second for one user with minimal wait. Batch 32 might give 600 tokens/second aggregate while each user waits longer to be scheduled. Total work rose; individual experience got worse. That is why throughput and latency are reported together — optimising one in isolation always degrades the other.

Commonly confused with

Throughput is not latency and improving one usually costs the other. Throughput is system-wide work per unit time; latency is what a single user waits. A serving setup can be excellent at one and unacceptable at the other.

When to use it

Reach for it when:

• Sizing capacity and cost per token for a workload
• Offline or batch jobs where nobody is waiting on an individual response
• Comparing serving configurations under a fixed latency budget

Reach for something else when:

• As the headline metric for interactive products — users feel latency, not throughput
• Quoted without batch size and context length, where the number means nothing
• As a proxy for cost; utilisation and hardware pricing decide that

Referenced by

This term is used by the following ModelRefs references:

• streaming inference stack
• batch inference stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Throughput — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path