Multi-Head Attention (MHA) — AI Glossary

Reference details

Related terms

• Attention Mechanism
• Grouped Query Attention (GQA)
• Cross-Attention

Example: Where the heads and the quadratic term come from

A model with hidden size 4,096 and 32 heads gives each head 4,096 / 32 = 128 dimensions. Each head computes attention independently over the sequence, and the results are concatenated back to 4,096 — so heads cost no extra width, they partition it. The cost that does grow is the pairwise comparison: at a sequence length of 8,192 each head scores 8,192² ≈ 67 million query-key pairs, and that term is quadratic in length while everything else is linear. Doubling the context roughly quadruples that work, which is the pressure behind every attention variant.

Commonly confused with

Multi-head attention is the original formulation. GQA and MQA are not different attention mechanisms but different sharing schemes for the key and value projections — they cut KV-cache memory at inference, not the quadratic score computation. Sliding-window and sparse attention are what actually attack the quadratic term, by not computing all pairs.

When to use it

Reach for it when:

• Reasoning about why long context costs what it does
• Interpreting a model card's head count and hidden size as a memory and bandwidth statement
• Understanding what GQA, MQA and windowed variants each trade away

Reach for something else when:

• Assuming more heads means better quality — head count partitions a fixed width
• Estimating inference memory from attention alone; the KV cache usually dominates
• Treating attention weights as an explanation of the model's reasoning

Primary source

• Vaswani et al., Attention Is All You Need (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Multi-Head Attention (MHA) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path