ModelRefs / Attention Mechanism — Tutorial
Attention Mechanism — Tutorial
The key innovation behind modern NLP — how models learn to focus on the right parts of their input. Covers The problem attention solves.
Overview
The key innovation behind modern NLP — how models learn to focus on the right parts of their input
Level: Advanced. Estimated reading time: 40 minutes.
The problem attention solves
Sequence-to-sequence models (encoder-decoder RNNs) compress the entire input into a single fixed-length vector — the encoder's final hidden state. For short sentences, this works. For long sentences (100+ words), this bottleneck loses information.
The intuition behind attention: when translating "The cat sat on the mat" to French, generating the word "chat" (cat) should focus on "cat" in the source, not the entire sentence. Instead of compressing everything into one vector, attention lets the decoder look at all encoder hidden states and selectively retrieve the relevant ones for each output step.
This solved the long-range dependency problem that plagued RNN encoder-decoders and enabled neural machine translation to approach human performance (Google Neural Machine Translation, 2016).
Query, Key, Value: the mechanics
Attention generalised into the Query-Key-Value framework:
- Query (Q): "what am I looking for?" — comes from the decoder's current state - Key (K): "what do I have?" — one per encoder hidden state - Value (V): "what to retrieve?" — the actual content to extract
Step 1: compute alignment scores: score(Q, Kᵢ) = Q · Kᵢ / √d_k. Dividing by √d_k prevents dot products from growing large and pushing softmax into saturation.
Step 2: softmax over scores → attention weights αᵢ (sum to 1, all ≥ 0)
Step 3: weighted sum of Values: context = Σ αᵢ · Vᵢ
The context vector is a soft retrieval — you don't pick one source word, you blend all of them weighted by relevance. This is fully differentiable, so attention weights are learned end-to-end.
From attention to self-attention
In the original (Bahdanau, 2015) attention, Q comes from the decoder and K/V come from the encoder — it is cross-attention between two sequences.
Self-attention: Q, K, V all come from the same sequence. Each position attends to every other position in the same sequence. This allows a word to gather context from all other words simultaneously, regardless of distance.
In matrix form: Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V
Where Q, K, V are matrices of shape (seq_len, d_k). The full computation is one matrix multiply + softmax + another matrix multiply. It is O(n²·d) in sequence length n — slower than RNNs (O(n·d²)) for very long sequences but parallelisable across all positions simultaneously.
Self-attention is the core operation of the Transformer architecture.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Attention Mechanism — Tutorial.