Group Normalization

Group normalization splits channels into groups and normalizes within each group per sample, giving batch-independent normalization for vision models.

Removing the batch dependence

Batch normalization normalizes each channel using statistics computed across the batch, which works well with large batches but degrades when batches are small, because the estimated mean and variance become noisy. Group normalization (GroupNorm) avoids this by computing statistics within a single sample. It divides a layer's channels into a fixed number of groups and normalizes the activations within each group, using the same mean and variance for every spatial location in that group.

The computation

For one sample, the channels are partitioned into G groups. Within each group, GroupNorm computes the mean and variance over all channels in the group and all spatial positions, normalizes those activations, and then applies a learned per-channel scale and shift. Because the statistics come from one sample, the result does not depend on batch size at all, so the same normalization behaves identically at training and inference and across any batch size.

• Batch-independent, so it is stable at batch size one
• Identical behavior in training and inference, with no running statistics to maintain
• The number of groups is a hyperparameter; the extremes recover other norms
• Well suited to detection, segmentation, and video where memory forces small batches

Relation to other norms

GroupNorm generalizes a family. With one group containing all channels it becomes layer normalization; with each channel in its own group it becomes instance normalization. Choosing an intermediate number of groups, often thirty-two, balances between these and tends to work best for convolutional vision networks. Unlike RMSNorm, which targets transformers, GroupNorm is designed for the channel structure of convolutional feature maps.

import torch.nn as nn
norm = nn.GroupNorm(num_groups=32, num_channels=256)
# statistics computed per-sample within each group of channels

When to use it

GroupNorm is the standard replacement for batch normalization when batches must be small, which is common in high-resolution vision and 3D tasks where memory is tight. It trades a small accuracy gap against very large-batch batch normalization for robustness across batch sizes and a simpler inference path. It is one of several normalization variants a designer selects based on the data structure and training regime.