ModelRefs / The Transformer Architecture — Tutorial
The Transformer Architecture — Tutorial
Positional encoding, layer norm, feed-forward sublayers — the full blueprint of modern LLMs. Covers The encoder block, Positional encoding.
Overview
Positional encoding, layer norm, feed-forward sublayers — the full blueprint of modern LLMs
Level: Advanced. Estimated reading time: 45 minutes.
The encoder block
The Transformer (Vaswani et al., 2017 — "Attention Is All You Need") replaced recurrence entirely with attention and feed-forward layers. One encoder block consists of:
1. Multi-head self-attention sublayer 2. Add & Norm: residual connection + layer normalisation 3. Position-wise feed-forward sublayer: two linear layers with a ReLU (or GELU) between them 4. Add & Norm again
The residual connections (x → x + sublayer(x)) allow gradients to flow directly from the output to each sublayer's input, enabling very deep stacking. The original paper used N=6 encoder blocks. GPT-3 uses 96.
Layer normalisation normalises across the feature dimension (not the batch dimension like BatchNorm): each token's embedding is normalised independently. This is critical for transformers because sequence lengths and batch sizes vary.
Positional encoding
Self-attention is permutation-equivariant — shuffling the input produces the same attention pattern, just shuffled. It has no notion of position. Positional encodings add position information to the input embeddings.
The original Transformer uses sinusoidal encodings: PE(pos, 2i) = sin(pos/10000^(2i/d_model)), PE(pos, 2i+1) = cos(pos/10000^(2i/d_model)). These are fixed (not learned) and generalise to sequence lengths longer than seen during training.
Modern LLMs use Rotary Position Embeddings (RoPE, used in Llama and Claude) or Alibi (used in some open models). These rotate the Q and K vectors based on position before the dot product, encoding relative distance more effectively than absolute sinusoidal encodings.
Decoder and encoder-decoder architectures
Three Transformer variants are in widespread use:
Encoder-only (BERT): processes the full input bidirectionally. [MASK] tokens are predicted during pretraining. Best for classification, NER, and text understanding tasks.
Decoder-only (GPT, Claude, Llama): autoregressive generation — predict the next token given all previous tokens. Uses causal masking. Best for generation tasks. Now the dominant paradigm for general-purpose LLMs.
Encoder-decoder (T5, BART): an encoder reads the input sequence, a decoder generates the output sequence attending to encoder states via cross-attention. Best for seq2seq tasks: translation, summarisation, question answering.
The feed-forward sublayer is the largest part by parameter count (d_model × d_ff × 2, typically d_ff = 4×d_model). In GPT-3, d_model=12288 and d_ff=49152 — about 3B parameters just in feed-forward layers across 96 blocks.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to The Transformer Architecture — Tutorial.