Latency — AI Glossary

Reference details

Related terms

• Inference
• Throughput
• Streaming (Token Streaming)

Example: Two numbers, two experiences

Time-to-first-token is when the user stops wondering whether it is working; tokens-per-second is how fast it then reads. A 200ms TTFT at 30 tok/s feels responsive. A 3-second TTFT at 90 tok/s feels broken for the first three seconds, no matter how fast it finishes. For streaming interfaces TTFT is the number users actually judge.

Commonly confused with

Latency is not throughput, and improving throughput usually worsens latency. Larger batches raise total tokens per second across all users while each individual request waits longer to be scheduled. A serving config can look excellent on one metric and unacceptable on the other.

When to use it

Reach for it when:

• The interface is interactive and a person is waiting
• Setting SLOs — TTFT and TPS need separate targets
• Comparing providers, where network distance can dominate model speed

Reach for something else when:

• Batch and offline jobs, where throughput is the metric that matters
• Quoted as an average — tail latency is what users complain about

Referenced by

This term is used by the following ModelRefs references:

• streaming inference stack
• edge runtime stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Latency — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path