ModelRefs / Cold Start — AI Glossary
Cold Start — AI Glossary
The latency spike when an inference container or GPU worker boots from idle, loading model weights from storage into VRAM.
Overview
Cold starts range from 1 s (cached container image) to 30+ s (loading 70B weights from S3). Mitigated by warm pools (keeping a minimum of idle replicas), model caching on local NVMe, and smaller models. Critical consideration for real-time user-facing applications using serverless inference.
Reference details
| Topic | inference |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Cold Start — AI Glossary.
Frequently asked questions
What is Cold Start?
The latency spike when an inference container or GPU worker boots from idle, loading model weights from storage into VRAM.
What concepts are related to Cold Start?
Closely related concepts include serverless inference, warm pool, gpu memory.