vLLM — AI Glossary

Reference details

Related terms

• PagedAttention
• Continuous Batching
• Inference

Example: Why continuous batching matters

With static batching, a batch of eight requests waits for the slowest to finish before any slot frees. Continuous batching evicts each sequence as it completes and admits a new one immediately, so the GPU is not idling on finished slots. That scheduling difference, not a faster kernel, is where most of the throughput gain comes from.

Commonly confused with

vLLM is an inference server, not a model and not a framework for building applications. It serves open-weight models over an OpenAI-compatible API. It does not fine-tune, does not orchestrate, and is not an alternative to a gateway — those sit at different layers.

When to use it

Reach for it when:

• Self-hosting open-weight models with real concurrency
• You need PagedAttention or LoRA hot-swap at serving time
• You want an OpenAI-compatible surface over your own hardware

Reach for something else when:

• You use hosted APIs — the provider already runs something like it
• Single-user or batch workloads, where a simpler runtime is easier to operate
• Nobody owns GPU operations; a serving stack is infrastructure to run

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to vLLM — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path