Prompt Caching — AI Glossary

Reference details

Related terms

• KV Cache
• Inference Cost
• System Prompt

Example: It pays for itself on the second request

Writing a prefix to the cache typically costs more than a normal input token — call it 1.25× — and reading it back costs a small fraction, call it 0.1×. Send a long shared prompt twice: uncached is 1.0 + 1.0 = 2.0, cached is 1.25 + 0.1 = 1.35. The break-even is inside the first repeat, and from there the saving compounds. The catch is the time-to-live: caches expire after minutes, so a low-traffic endpoint can pay the write premium every single time and never reach a read. Before enabling it, check the request interval against the TTL, not just the prompt size.

Commonly confused with

Prompt caching reuses computation for an identical prefix; semantic caching returns a stored *answer* for a similar question. The first is transparent and cannot change what the model says; the second can serve a stale or subtly wrong response and needs its own correctness thinking. Both differ from the KV cache within a single request, which never outlives it.

When to use it

Reach for it when:

• Long system prompts, few-shot blocks or documents repeated across many requests
• Agent loops and RAG, where a large fixed context is resent every step
• Traffic frequent enough that a cached prefix is read before it expires

Reach for something else when:

• Low-traffic endpoints, where the write premium is paid repeatedly for no read
• Prompts that vary early — the cache matches on a prefix, so put the variable parts last
• Assuming it changes output: it saves recomputation, nothing else

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Prompt Caching — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path