ModelRefs / Transformer — AI Glossary

Transformer — AI Glossary

The neural-network architecture using self-attention that powers virtually every modern LLM and multimodal AI model.

Overview

Introduced in 'Attention Is All You Need' (Vaswani et al., 2017). Decoder-only transformers dominate text generation (GPT, Claude, Llama). Encoder-decoder variants power translation and summarization. Vision transformers (ViT) apply the same architecture to image patches.

Reference details

Topic	architecture
Last reviewed	2026-06-24

Example: What self-attention buys

In “the trophy did not fit in the suitcase because it was too small”, resolving “it” requires weighing every other word. Recurrent models pass information along a chain and degrade with distance; self-attention lets every token attend to every other directly, in one step. The cost is that attention is quadratic in sequence length, which is why long context is expensive.

Commonly confused with

Transformer is the architecture, not the model. GPT, Claude, Llama and BERT are all transformers but differ in pretraining, scale and whether they are decoder-only, encoder-only or encoder-decoder. Saying a system “uses a transformer” says almost nothing about what it can do.

When to use it

Reach for it when:

It is the default for sequence modelling; the real choice is which variant and size
Long-range dependencies matter and recurrence would lose them
You need parallel training over sequences rather than step-by-step

Reach for something else when:

Sequences are extremely long and quadratic attention dominates — state-space models target this
The task is tabular or small-data, where simpler models win on cost and interpretability

Primary source

Vaswani et al., Attention Is All You Need (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Transformer — AI Glossary.

Frequently asked questions

What is Transformer?