Momentum

Accumulate a velocity vector from past gradients to accelerate along consistent directions and damp oscillations across valleys.

The heavy-ball idea

Plain gradient descent oscillates in narrow, curved valleys because the gradient keeps pointing across the valley rather than down it. Momentum, also called the heavy-ball method, adds inertia: it maintains a velocity that blends the previous velocity with the current gradient, so consistent components accumulate and oscillating components cancel.

The update

v_{k+1} = b * v_k - a * grad f(x_k); x_{k+1} = x_k + v_{k+1}. Here b in [0,1) is the momentum coefficient (commonly 0.9) and a is the step size. When b = 0 this reduces to gradient descent. The velocity is an exponentially weighted moving average of past gradients.

Why it accelerates

On a quadratic with condition number kappa, gradient descent needs about kappa iterations to reach a target accuracy, while well-tuned momentum needs about sqrt(kappa). This is a large gain for ill-conditioned problems. Intuitively, momentum builds speed along the low-curvature direction of a valley while averaging out the high-curvature cross oscillations.

Choosing the coefficient

• b too small: little acceleration, behaves like gradient descent.
• b too large: overshoot and instability, especially with a large step size.
• 0.9 to 0.99 is typical; larger b usually pairs with a smaller step size.

Physical analogy

Think of a ball rolling down the loss surface with friction. The gradient is the force, b controls how much momentum carries over (low friction means b near 1). The ball rolls through small bumps and shallow local dips that would trap a memoryless descent.

v = 0
for _ in range(iters):
    v = beta*v - lr*grad(x)
    x = x + v

Momentum, especially the Nesterov variant, is a standard ingredient in training deep models and in accelerated first-order solvers for engineering design.