Rotary Position Embeddings
Rotary position embeddings encode position by rotating query and key vectors, so attention depends on the relative distance between tokens.
Position through rotation
Transformers need a way to encode token order, since attention is otherwise order-blind. Rotary position embeddings (RoPE) do this by rotating each pair of dimensions in the query and key vectors by an angle proportional to the token's position. Different dimension pairs rotate at different frequencies, from fast to slow, so together they encode position across many scales. Crucially, the rotation is applied to the queries and keys before the attention dot product, not added to the token embeddings.
Why relative position emerges
The dot product between a rotated query at position m and a rotated key at position n depends only on their difference m minus n, because rotating both by their respective angles and taking the inner product leaves a function of the relative angle. This means RoPE gives the model relative position information for free within the standard attention operation, combining the simplicity of absolute encodings with the generalization benefits of relative ones.
# apply rotation to pairs of dimensions at position m
theta = 1.0 / (10000 ** (torch.arange(0, d, 2)/d))
angles = m * theta
q_rot = q_even*cos(angles) - q_odd*sin(angles) # rotate each pair
k_rot = k_even*cos(angles) - k_odd*sin(angles)- Injects relative position directly into the attention score
- No learned position table, so it is parameter-free
- Rotations preserve vector norms, keeping magnitudes stable
- Extends to longer sequences than seen in training, especially with frequency scaling
Length extrapolation
Because position enters as a continuous rotation, RoPE can be adapted to context lengths beyond training by scaling the rotation frequencies, for example stretching the effective positions so a longer sequence maps onto the range the model already understands. This makes RoPE a practical choice for models that must handle variable and growing context windows. It has largely replaced additive learned positions like those in ViT in modern decoder-only models.