ModelRefs / Chunked Prefill — AI Glossary
Chunked Prefill — AI Glossary
An inference optimization splitting long prompt prefill into chunks to interleave compute with token generation, reducing time-to-first-token.
Overview
Prefill (computing KV cache for the prompt) and decode (generating tokens) have different compute profiles. Chunked prefill (vLLM 0.4+) breaks large prefill into chunks interleaved with ongoing decode requests, preventing long prompts from stalling generation for other requests. Improves p99 TTFT under high load.
Reference details
| Topic | inference |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Chunked Prefill — AI Glossary.
Frequently asked questions
What is Chunked Prefill?
An inference optimization splitting long prompt prefill into chunks to interleave compute with token generation, reducing time-to-first-token.
What concepts are related to Chunked Prefill?
Closely related concepts include time to first token, kv cache, batch size.