ALiBi (Attention with Linear Biases) — AI Glossary

Reference details

Related terms

• Positional Encoding
• RoPE (Rotary Positional Embedding)
• Long-Context Model

Example: A penalty proportional to distance, learned by nobody

Instead of adding position information to the inputs, this subtracts a penalty from the attention score itself, proportional to how far apart two tokens are, with a fixed per-head slope. On a head with slope 1/2, a gap of 100 tokens costs 50 and a gap of 200 costs 100. Attention therefore decays with distance without ever being cut off. Because no position is ever learned, nothing breaks when the sequence runs longer than anything seen in training — the penalty is simply computed for the larger gap. That is the property the method was built for, and it is a different mechanism from rotating the query and key vectors.

Commonly confused with

All three positional schemes are commonly lumped together and behave differently. Learned absolute embeddings add a vector per position at the input and cannot extrapolate past the trained length. Rotary encoding rotates query and key vectors so scores depend on relative offset. ALiBi adds a distance-proportional bias to the scores. Only the last two extrapolate by construction, and modern open-weights models overwhelmingly chose rotation.

When to use it

Reach for it when:

• Reading or reproducing models from the era that adopted it
• Understanding why extrapolation behaviour differs so much between architectures
• Settings where a monotonic recency bias is a desirable inductive bias

Reach for something else when:

• New architectures, where rotation plus a context-extension method is the current default
• Tasks needing strong attention to distant tokens, which the penalty works against
• Assuming extrapolation is unlimited — quality still degrades well beyond training length

Primary source

• Press et al., Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to ALiBi (Attention with Linear Biases) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path