ModelRefs / KV Cache — AI Glossary

KV Cache — AI Glossary

Cached key/value tensors from all previous tokens that let the transformer generate each new token without recomputing earlier positions.

Overview

KV cache is the dominant GPU memory cost at long contexts — it grows linearly with sequence length. PagedAttention, prefix caching, and quantized KV cache all attack this. Multi-query attention (MQA) and grouped-query attention (GQA) reduce KV cache size by sharing heads.

Reference details

Topic	inference
Last reviewed	2026-06-24

Example: Why long contexts get expensive fast

Without the cache, generating token 5,000 means recomputing attention over the preceding 4,999 — generation becomes quadratic and unusable. The cache stores those keys and values so each new token attends to stored state. The cost moves from compute to memory, and that memory grows linearly with context, which is why long-context serving is a memory problem before it is a speed problem.

Commonly confused with

The KV cache is not prompt caching. The KV cache lives for a single sequence and holds per-token tensors in GPU memory. Prompt caching is a provider feature that reuses a shared prefix across separate requests, usually for billing and latency, and is a different mechanism at a different layer.

When to use it

Reach for it when:

It is on by default in every serious serving stack — the decision is how to size it
Reason about it when planning long-context or high-concurrency capacity
Consider quantized KV or GQA when it dominates your memory budget

Reach for something else when:

Do not treat it as free: at long context it can exceed the weights themselves
Do not assume concurrency scales with GPU memory — cache per request bounds it

Referenced by

This term is used by the following ModelRefs references:

become llm engineer

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to KV Cache — AI Glossary.

Frequently asked questions

What is KV Cache?