The Hessian and Curvature
The Hessian matrix of second derivatives describes an objective's local curvature, classifying critical points and governing how fast optimizers converge.
The matrix of second derivatives
For a function f of several variables, the Hessian H is the matrix of all second partial derivatives, H_ij = d^2 f / (dx_i dx_j). It is symmetric for smooth functions. Where the gradient tells you the slope, the Hessian tells you how the slope changes, that is, the local curvature. The second-order Taylor expansion f(x + p) approximately f(x) + grad^T p + (1/2) p^T H p uses the Hessian as its quadratic term.
Classifying critical points
At a critical point where the gradient is zero, the Hessian's eigenvalues determine the point's type. All eigenvalues positive (H positive definite) means a local minimum; all negative means a local maximum; mixed signs mean a saddle point, downhill in some directions and uphill in others. A zero eigenvalue leaves the test inconclusive. This second-derivative test is the multidimensional generalization of checking whether a curve is concave up or down.
- Positive definite Hessian: strict local minimum
- Negative definite Hessian: strict local maximum
- Indefinite Hessian (mixed signs): saddle point
- Singular Hessian: test inconclusive, higher-order info needed
Curvature and convergence speed
The Hessian's eigenvalues set the scale of the landscape in each direction: a large eigenvalue is a steep, high-curvature direction, a small one a shallow direction. Gradient descent's convergence rate depends on the ratio of largest to smallest eigenvalue, the condition number. When curvature is very different across directions, gradient descent zig-zags and slows. Newton's method multiplies the gradient by the inverse Hessian to rescale every direction, achieving fast local convergence.
Using it without forming it
The full Hessian costs memory quadratic in the number of variables and is expensive to invert, so large-scale methods avoid forming it. Quasi-Newton methods (BFGS) build a low-cost approximation from gradient differences; Hessian-free methods use only Hessian-vector products, which can be computed at the cost of one extra gradient evaluation, inside a conjugate-gradient solve. Curvature information is what separates fast second-order methods from slow first-order ones.