Backpropagation
The algorithm that computes a neural network's loss gradients efficiently by applying the chain rule backward through layers.
Definition
Backpropagation computes the gradient of the loss with respect to every weight in a network. It applies the chain rule of calculus, propagating error signals from the output layer back toward the input, reusing intermediate results so the whole gradient costs about as much as one forward pass.
Because it is exact and efficient, backpropagation removed the guesswork from training deep models: gradients tell each parameter precisely how to change to reduce error. The same reverse-mode differentiation now underpins scientific computing beyond machine learning, wherever sensitivities of an output to many inputs are needed.
Its efficiency comes from reusing the forward pass's stored activations, which is why memory can become the binding constraint for very deep networks. Techniques such as gradient checkpointing trade extra computation for reduced memory by recomputing activations on demand. Understanding this compute-versus-memory trade-off is essential to training large models on limited hardware.
The gradients it produces feed an optimizer such as stochastic gradient descent, which updates the weights.
How it works
- Forward pass: compute activations layer by layer.
- Compute the loss at the output.
- Backward pass: propagate partial derivatives back through each layer.
- Accumulate the gradient for every parameter.
Why it matters
Backpropagation is what makes deep networks trainable at scale. It is a special case of reverse-mode automatic differentiation, the same technique that powers modern differentiable programming frameworks.
Fusion connection
The differentiability that backpropagation exploits also lets Kronos propagate gradients through surrogate models, enabling gradient-based search over machine parameters rather than blind sampling.