Attention Mechanism — AI Glossary

Reference details

Related terms

• Transformer
• KV Cache
• FlashAttention

Example: Why doubling context quadruples the cost

Every token attends to every other, so the attention matrix has n² entries. At 1,000 tokens that is 1,000,000 pairs; at 2,000 it is 4,000,000. Double the input, quadruple the attention work. That single fact drives most long-context engineering: sliding windows, grouped-query attention and FlashAttention all exist to bend that curve.

n =  1,000 -> n^2 =     1,000,000 pairs
n =  2,000 -> n^2 =     4,000,000 pairs   # 2x input, 4x work
n = 32,000 -> n^2 = 1,024,000,000 pairs

Commonly confused with

Attention weights are not explanations. A high weight between two tokens shows where the computation looked, not why the output is what it is — attention maps have repeatedly been shown to be unreliable as interpretability evidence.

When to use it

Reach for it when:

• Reasoning about why long context is expensive rather than merely limited
• Choosing between attention variants when memory is the constraint
• Understanding why prefill cost scales worse than generation cost

Reach for something else when:

• As an explanation of model behaviour for an audit or a user
• Assuming all modern models use full quadratic attention — many do not

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Attention Mechanism — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path