The Wolfe Conditions
The Wolfe conditions are two inequalities a line-search step must satisfy to guarantee both sufficient decrease and enough curvature progress.
Two requirements on a step
For a line search along direction p from point x, a good step length alpha must avoid two failures: being too long, so it overshoots and the decrease is small relative to the distance, and being too short, so almost no progress is made. The Wolfe conditions are a pair of inequalities that exclude both, and satisfying them lets convergence be proven for gradient and quasi-Newton methods.
The sufficient-decrease condition
The first Wolfe condition, the Armijo rule, requires f(x + alpha p) <= f(x) + c_1 alpha grad_f(x)^T p, with 0 < c_1 < 1 (typically 1e-4). It demands that the actual decrease be at least a fraction c_1 of the decrease predicted by the linear model. This prevents steps that are too long by insisting the function actually falls in line with the downhill slope.
The curvature condition
The second Wolfe condition requires grad_f(x + alpha p)^T p >= c_2 grad_f(x)^T p, with c_1 < c_2 < 1 (often 0.9). It says the slope along p at the new point must be less steep (less negative) than it was, meaning the step went far enough that the function is flattening out. This prevents steps that are too short. The strong Wolfe condition tightens this to |grad_f(x + alpha p)^T p| <= c_2 |grad_f(x)^T p|, keeping the new point near a one-dimensional minimum.
- c_1 controls sufficient decrease (small, e.g. 0.0001)
- c_2 controls the curvature requirement (larger, e.g. 0.9)
- Require c_1 < c_2 < 1 for the conditions to be satisfiable
- Strong Wolfe bounds the magnitude, not just the sign, of the new slope
Why they are needed
The Wolfe conditions guarantee that a step exists (for any smooth function bounded below along the direction) and that a method taking Wolfe-satisfying steps converges globally, via the Zoutendijk theorem. They are essential to quasi-Newton methods such as BFGS for a second reason: the curvature condition ensures the update to the approximate Hessian keeps it positive definite, so the next search direction is genuinely downhill. A line search enforcing strong Wolfe is the standard companion to BFGS and L-BFGS.