Positional Encoding

Because attention ignores order, transformers add positional encodings so the model knows where each token sits in the sequence.

Why order must be injected

Self-attention treats its input as a set: shuffle the tokens and the attention computation gives the same set of outputs, just reordered. Language and most sequences depend on order, so a transformer needs an explicit signal of position. Positional encoding supplies each position with a distinct vector that is combined with the token embedding, giving the model access to both content and location.

Sinusoidal encoding

Kronos motion — pid vs model

The original transformer used fixed sinusoids: each dimension of the position vector is a sine or cosine of the position at a different frequency. Low frequencies vary slowly across positions and high frequencies vary quickly, so the combination uniquely identifies each position. A useful property is that the encoding of a position can be expressed as a linear function of another, which helps the model reason about relative offsets and generalize to lengths not seen in training.

python

import numpy as np
def sinusoidal(pos, d):
    pe = np.zeros((pos, d))
    for i in range(0, d, 2):
        f = 1 / 10000 ** (i / d)
        pe[:, i]   = np.sin(np.arange(pos) * f)
        pe[:, i+1] = np.cos(np.arange(pos) * f)
    return pe

Learned position embeddings

An alternative treats each position as a token in a learnable lookup table, training a position embedding just as word embeddings are trained. This is simple and effective but caps the model at the maximum length seen in training, since positions beyond that have no learned vector. Many early large models, including the original BERT and GPT lines, used learned position embeddings.

Relative and rotary encodings

Newer schemes encode the distance between tokens rather than absolute positions, which often generalizes better to longer sequences. Rotary positional embedding (RoPE) rotates query and key vectors by an angle proportional to position, so their dot product naturally depends on relative offset. RoPE and related relative schemes are now common in large language models.

Absolute versus relative

Absolute encodings tell the model where a token is; relative encodings tell it how far apart two tokens are. Relative information is often what matters for language structure, and relative or rotary methods tend to extrapolate better past the training length. The choice affects a model's ability to handle inputs longer than those it was trained on.

Attention is order-blind, so position must be added.
Sinusoidal encodings are fixed and extrapolate somewhat.
Learned embeddings are simple but length-capped.
Rotary and relative schemes encode distance directly.