ModelRefs / Serverless Inference — AI Glossary
Serverless Inference — AI Glossary
A deployment model where inference capacity scales to zero when idle and scales up automatically per request, billed per-token. Also called on-demand inference.
Overview
Serverless LLM inference (AWS Bedrock, Together AI, Modal, Replicate) eliminates idle GPU cost by spinning down between requests. Cold starts (loading weights) add 1–10 s latency. Ideal for bursty or low-traffic workloads. Contrast with dedicated endpoints (always-on GPU, predictable latency, higher baseline cost).
Reference details
| Topic | inference |
|---|---|
| Also known as | on-demand inference |
| Last reviewed | 2026-06-24 |
Related terms
Example: Where the break-even sits
A dedicated endpoint bills continuously whether or not traffic arrives; serverless bills per token but adds a cold start of seconds when scaled to zero. For bursty or low-volume workloads serverless is far cheaper. For steady high traffic the dedicated endpoint wins on both cost and latency — the crossover is a traffic-shape question, not a preference.
Commonly confused with
Serverless does not mean no cold start. Scaling to zero is precisely what creates one: multi-gigabyte weights have to be loaded before the first token. Providers mitigate it with warm pools, but the trade-off is inherent to scaling to zero.
When to use it
Reach for it when:
- Traffic is bursty, unpredictable or low-volume
- Idle GPU cost is the dominant line item
- A first-request delay is acceptable
Reach for something else when:
- Interactive products where a cold start lands on a real user
- Steady high throughput, where dedicated capacity is cheaper per token
- Strict tail-latency SLOs
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Serverless Inference — AI Glossary.
Frequently asked questions
What is Serverless Inference?
A deployment model where inference capacity scales to zero when idle and scales up automatically per request, billed per-token.
Is Serverless Inference the same as on-demand inference?
Yes — on-demand inference are common aliases for Serverless Inference.
What concepts are related to Serverless Inference?
Closely related concepts include cold start, warm pool, inference provider.