Backpropagation
Backpropagation computes a neural network's loss gradients efficiently by applying the chain rule backward through the layers.
The chain rule, organized
Backpropagation is the algorithm that computes the gradient of a neural network's loss with respect to every weight, efficiently. A network is a composition of layers; the gradient of the loss with respect to an early weight requires the chain rule through all later layers. Backpropagation organizes that computation so the whole gradient costs about the same as one forward pass.
Two passes
- Forward pass: feed the input through the layers, computing and storing each layer's output.
- Backward pass: start from the loss and propagate the gradient backward, layer by layer, multiplying by each layer's local derivative.
- Update: feed the resulting weight gradients to an optimizer such as SGD.
The efficiency comes from reuse: intermediate gradients computed for a layer are shared by all the weights feeding it, so nothing is recomputed. This is dynamic programming applied to the chain rule.
Why activations must store state
Each layer's local derivative depends on its inputs and outputs from the forward pass, so those values are cached. This is why training uses more memory than inference. The backward pass multiplies these local derivatives together, which is also why very deep networks suffer vanishing or exploding gradients when the factors are consistently small or large.
# conceptual: one dense layer with sigmoid
# forward: z = X@W + b ; a = sigmoid(z)
# backward: dz = da * a*(1-a)
# dW = X.T @ dz ; dX = dz @ W.TAutomatic differentiation
Modern frameworks implement backpropagation as reverse-mode automatic differentiation: you specify the forward computation and the framework derives the exact gradients automatically. This frees practitioners to design architectures without deriving gradients by hand. The gradients then drive gradient descent to train the network.