Natural Gradient
The natural gradient rescales the ordinary gradient by the Fisher information matrix, following the steepest descent direction in distribution space.
The problem with the ordinary gradient
When parameters index a family of probability distributions, the ordinary gradient measures steepest descent in raw parameter space, which is not meaningful: the same distribution can be parameterized many ways, and Euclidean distance between parameters does not reflect distance between distributions. A small parameter change can shift the distribution a lot, or barely at all.
The Fisher metric
The natural gradient uses the Fisher information matrix F as the local metric on distribution space. The Fisher matrix is the expected outer product of the score (the gradient of the log-likelihood) and equals the second-order (KL-divergence) curvature of the distribution family. The natural-gradient direction is F^{-1} grad L, the ordinary gradient preconditioned by the inverse Fisher matrix.
This direction is the steepest descent when distance is measured by KL divergence rather than Euclidean distance, so it is invariant to how the model is parameterized. Reparameterize the model and the natural-gradient trajectory in distribution space is unchanged, which is not true of ordinary gradient descent.
Cost and approximation
The obstacle is that F is a full matrix in the number of parameters, so forming and inverting it is infeasible for large models. Practical methods approximate it: Kronecker-factored curvature (K-FAC) factors F blockwise into small Kronecker products; conjugate gradient solves F v = grad without materializing F using only Fisher-vector products; diagonal approximations underlie some adaptive optimizers.
- Exact Fisher: correct but O(p^2) storage and O(p^3) inversion
- K-FAC: block Kronecker factorization, tractable per layer
- Truncated conjugate gradient: matrix-free, needs Fisher-vector products
- Empirical Fisher: uses observed gradients, a cheaper surrogate
Where it is used
Natural gradient is standard in policy-gradient reinforcement learning (natural policy gradient, TRPO) and in variational inference, where following the KL geometry stabilizes training and improves sample efficiency. It sits alongside mirror descent as a way of choosing a problem-adapted geometry rather than defaulting to Euclidean steps.