Weight Initialization
Weight initialization sets the starting values of parameters so that signals and gradients keep a stable scale through a deep network at the first step.
Why the starting point matters
Before training begins, weights must be set to some values. Setting them all to zero is fatal, because every neuron in a layer would compute the same thing and receive the same gradient, so they could never differentiate. Random values break this symmetry, but the scale of that randomness is critical. If weights are too large, activations grow layer by layer and explode; too small, and they shrink toward zero. Either extreme makes gradients vanish or explode, stalling training in a deep network.
Variance-preserving schemes
Principled initializations choose the variance of the random weights so that the variance of activations stays roughly constant as signals pass forward, and gradients stay stable passing backward. Xavier (Glorot) initialization sets the variance based on the number of inputs and outputs of a layer, appropriate for symmetric nonlinearities like tanh. He (Kaiming) initialization scales by the number of inputs and accounts for the fact that a rectified linear unit zeroes out about half its inputs, so it doubles the variance to compensate. He initialization is the standard for ReLU-family networks.
import math, torch
# He (Kaiming) normal for a ReLU layer with fan_in inputs
std = math.sqrt(2.0 / fan_in)
W = torch.randn(fan_out, fan_in) * std- Xavier: variance from fan-in and fan-out, for tanh and similar
- He: variance from fan-in with a factor of two, for ReLU
- Biases usually start at zero
- Orthogonal initialization keeps signal norms exactly preserved, useful for deep or recurrent stacks
Interaction with modern designs
Good initialization matters most in the first steps of training; once normalization and residual connections are in place, they actively maintain signal scale and reduce sensitivity to the exact initialization. Even so, very deep residual networks benefit from initializing the residual branches small, so the network starts close to an identity map and gradually learns to deviate. Combined with a warmup schedule, careful initialization keeps training stable from the very first update.