Multi-Head Attention

Running several attention operations in parallel lets a model attend to different kinds of relationships at once, then combine them.

Why multiple heads

A single attention operation produces one weighted blend per position, which forces all relationships to share one pattern of focus. Multi-head attention runs several attention operations, called heads, in parallel, each with its own learned query, key, and value projections. Different heads can specialize: one may track syntactic dependencies, another may follow coreference, another may attend locally. The model gains a richer view of relationships than any single head could provide.

The construction

Kronos motion — pid vs model

The model splits the representation into h heads of smaller dimension d/h. Each head projects the input to its own Q, K, V and performs scaled dot-product attention independently. The h outputs are concatenated back to the full dimension and passed through a final linear projection that mixes information across heads. Total computation is similar to one full-size attention because each head is proportionally smaller.

python

# per head: project to d/h, attend, then concatenate
# outputs: [head_1; head_2; ...; head_h] @ W_out
# each head_i = attention(X @ Wq_i, X @ Wk_i, X @ Wv_i)

What heads learn

Analysis of trained transformers shows heads that specialize in interpretable functions: attending to the previous token, to matching brackets, to the subject of a verb, or to rare tokens. Many heads are redundant, and some can be pruned with little loss. Specialization is emergent, not designed; the architecture only provides the capacity for it.

Dimensions and cost

A common configuration uses a model dimension of a few hundred to a few thousand split into 8 to 96 heads. Because heads run in parallel and each is small, the added flexibility comes at modest cost. The concatenation and output projection ensure the combined result has the same shape as the input, so multi-head attention slots cleanly into a residual stream.

Role in the transformer

Multi-head attention is the core sublayer of every transformer block, used for self-attention within a sequence and for cross-attention between encoder and decoder. Wrapping it in residual connections and layer normalization, and alternating with position-wise feedforward layers, produces the full transformer that underlies modern language and vision models.

Several attention heads run in parallel with separate projections.
Heads specialize in different relationships.
Outputs are concatenated and projected back.
Core sublayer of every transformer block.