RoPE (Rotary Positional Embedding) — AI Glossary

Reference details

Related terms

• Positional Encoding
• ALiBi (Attention with Linear Biases)
• Long-Context Model

Example: Why relative position is the useful property

RoPE rotates the query and key vectors by an angle proportional to their position. Because the dot product of two rotated vectors depends only on the difference of the two angles, the attention score between positions 100 and 105 has the same geometry as between 1,000 and 1,005 — the model sees “five apart”, not “at 1,000”. That is what makes context extension tractable: methods that rescale or interpolate the rotation frequencies stretch the same learned relative structure over a longer window, instead of asking the model to handle absolute positions it never saw in training.

Commonly confused with

RoPE encodes relative position by rotation inside attention; learned absolute embeddings add a per-position vector at the input; ALiBi adds a distance-proportional penalty to attention scores. All three are “positional encoding”, and only the latter two are added to anything — RoPE modifies the vectors in place, which is why it composes with attention kernels so cleanly.

When to use it

Reach for it when:

• Designing or fine-tuning a decoder that may need to run beyond its training length
• Reasoning about why a context-extension technique works and what it costs
• Interpreting a model card that lists a base frequency — it governs the extension headroom

Reach for something else when:

• Assuming extension is free: stretching the window degrades quality without adaptation or continued training
• Comparing context lengths across models as if the mechanism were identical
• As an explanation for retrieval failures inside the window, which are an attention problem

Primary source

• Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to RoPE (Rotary Positional Embedding) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path