ModelRefs / Inference — AI Glossary
Inference — AI Glossary
The process of running a trained model to produce outputs from inputs at deployment time. Inference is distinct from training.
Overview
Inference is distinct from training. It dominates production cost for most AI products and is optimized via quantization, batching, KV-cache reuse, speculative decoding, and continuous batching. GPU/accelerator memory bandwidth is usually the binding constraint.
Reference details
| Topic | inference |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Example: Two phases with different bottlenecks
Prefill processes the whole prompt in parallel and is compute-bound — it is why time-to-first-token scales with prompt length. Decode then emits one token at a time, each requiring a pass over the KV cache, and is memory-bandwidth-bound. Optimisations target one phase or the other, which is why a change that improves throughput can leave latency untouched.
Commonly confused with
Inference is not the inverse of training. Training computes gradients over batches; inference is forward passes only, dominated by memory bandwidth rather than raw compute. Hardware that trains well does not automatically serve well, and the reverse also holds.
When to use it
Reach for it when:
- Reasoning about production cost, which inference dominates for most products
- Choosing serving hardware, where bandwidth usually binds before FLOPs
- Diagnosing whether a slowdown is prefill or decode
Reach for something else when:
- Do not size capacity from training benchmarks; the constraints differ
- Do not treat throughput and latency as one number — they optimise against each other
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Inference — AI Glossary.
Frequently asked questions
What is Inference?
The process of running a trained model to produce outputs from inputs at deployment time.
What concepts are related to Inference?
Closely related concepts include latency, throughput, quantization, inference cost.