Control Theory

The Hamilton-Jacobi-Bellman Equation

The HJB equation is the dynamic-programming condition for optimal control, defining the value function whose gradient yields the optimal feedback law.

Optimality Over the Whole State Space

The Hamilton-Jacobi-Bellman (HJB) equation expresses Bellman's principle of optimality as a partial differential equation. Where Pontryagin's principle characterizes a single optimal trajectory, the HJB equation characterizes the optimal cost from every possible state, producing a feedback law valid everywhere rather than a single open-loop plan.

The value function

Kronos motion — control room

Define the value function V(x, t) as the minimum cost achievable starting from state x at time t. The HJB equation states that at each point, the time rate of change of V plus the minimum over the input of the Hamiltonian equals zero. Solving this PDE for V yields the optimal cost, and the minimizing input at each state is the optimal feedback control.

Bellman's principle

The equation follows from the principle of optimality: any tail of an optimal trajectory is itself optimal for the subproblem starting where the tail begins. This lets a long-horizon problem be broken into an instantaneous decision plus the optimal value of the remainder, which is exactly what the HJB equation encodes.

Strengths and the curse of dimensionality

Sufficiency: a solution to the HJB equation guarantees optimality, whereas Pontryagin gives only necessary conditions.
Feedback: the solution is a feedback law over the whole state space, robust to where the system actually is.
Curse of dimensionality: solving the PDE requires gridding the state space, whose size grows exponentially with the number of states, making exact solutions infeasible for high-order systems.

The linear-quadratic special case

For linear dynamics and quadratic cost, the value function is quadratic in the state and the HJB equation collapses to the algebraic Riccati equation, whose solution gives the LQR gain. This is the rare case where the HJB equation is solved exactly and cheaply, and it explains why LQR sits at the center of optimal control.

For nonlinear or high-dimensional problems the HJB equation is approximated, by reinforcement learning, approximate dynamic programming, or online replanning as in MPC, all of which trade exact optimality for tractability.