Layer Normalization

Layer normalization standardizes across features within a single example, making it independent of batch size and ideal for transformers.

Normalizing per example

Layer normalization computes the mean and variance across all the features of a single training example and normalizes that example to zero mean and unit variance, then applies a learnable scale and shift. Crucially, it does not use any information from other examples in the batch. This makes it independent of batch size, in sharp contrast to batch normalization, which pools statistics across the batch.

Batch norm versus layer norm

Kronos motion — synchrotron size

The two differ in which axis they normalize. Batch norm normalizes each feature across the batch dimension; layer norm normalizes each example across the feature dimension. As a result batch norm couples examples together and needs a reasonable batch size, while layer norm treats each example alone and works with a batch of one. For sequence data of varying length, layer norm sidesteps the padding and length issues that trouble batch norm.

python

import numpy as np
def layer_norm(x, gamma, beta, eps=1e-5):
    mu = x.mean(axis=-1, keepdims=True)
    var = x.var(axis=-1, keepdims=True)
    xhat = (x - mu) / np.sqrt(var + eps)
    return gamma * xhat + beta

Why transformers use it

Transformers process variable-length sequences and are often trained with small effective batches per device, both awkward for batch norm. Layer norm's per-example operation fits naturally and behaves identically at training and inference, with no running statistics to track. Every attention and feedforward sublayer in a transformer is wrapped with layer normalization, applied either before the sublayer (pre-norm, now standard for stability) or after it (post-norm, in the original design).

Variants

RMSNorm simplifies layer norm by normalizing only by the root-mean-square of the features, dropping the mean subtraction and often the bias, which is slightly cheaper and works well in large language models. Group normalization, which normalizes over groups of channels, is a middle ground used in vision when batches are small. The common thread is normalizing without depending on other examples in the batch.

Effect on training

Like batch norm, layer norm stabilizes and speeds training by keeping activations well scaled and improving gradient behavior through deep stacks. Its independence from batch composition also makes results more reproducible, since a given example's normalization does not depend on which other examples share its batch. This reliability is part of why it became the default for large-scale sequence models.

Normalizes across features within one example.
Independent of batch size, unlike batch norm.
No train-test statistics mismatch.
The standard normalization inside transformers.