ModelRefs / Time to First Token (TTFT) — AI Glossary
Time to First Token (TTFT) — AI Glossary
The latency from sending a request until the first output token is received; the primary UX latency metric for streaming responses.
Overview
TTFT includes network RTT + prefill compute (processing the entire prompt). Long prompts increase TTFT proportionally. Streaming APIs emit tokens as they are generated, so TTFT governs perceived responsiveness. Disaggregated serving and chunked prefill reduce TTFT under load. Typical targets: <500 ms for interactive, <200 ms for real-time.
Reference details
| Topic | inference |
|---|---|
| Also known as | TTFT, first token latency |
| Last reviewed | 2026-06-24 |
Related terms
Example: It scales with the prompt, not the answer
Before any token comes back, the model must process the entire prompt — the prefill pass. A 500-token prompt and a 50,000-token prompt differ by two orders of magnitude in that work, so the long one waits proportionally longer for its first token even if both answers are three sentences. This is why stuffing a document into context is felt as sluggishness rather than cost, why prompt caching improves responsiveness so noticeably (a cached prefix skips most of the prefill), and why trimming a system prompt often does more for perceived speed than switching to a faster model.
Commonly confused with
Time to first token is the wait before output starts; tokens per second is how fast it flows once it has. They are governed by different things — prefill and queueing for the first, memory bandwidth and batch size for the second — and are optimised by different changes. Streaming exposes TTFT directly to the user, so it becomes the number that reads as “speed”.
When to use it
Reach for it when:
- Interactive and voice interfaces, where the pause before output is what users judge
- Deciding whether to shorten prompts, cache prefixes, or move to retrieval
- Capacity planning under load, since queueing lands here first
Reach for something else when:
- Batch and background work, where nobody is waiting on the first token
- Comparing providers without matching prompt length and cache state
- As a total-latency proxy — a long answer's completion time is dominated by decode
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Time to First Token (TTFT) — AI Glossary.
Frequently asked questions
What is Time to First Token (TTFT)?
The latency from sending a request until the first output token is received; the primary UX latency metric for streaming responses.
Is Time to First Token (TTFT) the same as TTFT?
Yes — TTFT, first token latency are common aliases for Time to First Token (TTFT).
What concepts are related to Time to First Token (TTFT)?
Closely related concepts include tokens per second, p99 latency, chunked prefill.