The Bellman Equation

The Bellman equation expresses a value as immediate reward plus the discounted value of what follows, making long-horizon value recursive.

Value defined recursively

The Bellman equation is the central identity of reinforcement learning. It states that the value of a state equals the expected immediate reward plus the discounted value of the next state. This recursion is what lets an agent reason about long futures one step at a time.

For state values

Under policy pi, V^pi(s) = sum over a of pi(a | s) sum over s' of P(s' | s, a) [ R(s, a, s') + gamma V^pi(s') ]. The value of now is the reward you expect plus a discounted echo of the value of next. Because V^pi appears on both sides, the equation is a fixed-point condition the true value function must satisfy.

For action values

The action-value form is Q^pi(s, a) = sum over s' of P(s' | s, a) [ R(s, a, s') + gamma sum over a' of pi(a' | s') Q^pi(s', a') ]. This version underlies temporal-difference control methods such as SARSA and Q-learning.

Bellman as an operator

The right-hand side defines the Bellman operator. Applying it repeatedly to any initial guess drives the estimate toward the true value function because the operator is a contraction under discounting: each application shrinks the error by a factor gamma. This convergence guarantee is the foundation of dynamic programming.

Why it is central

Nearly every RL algorithm is a way of solving or approximating a Bellman equation. Exact methods sweep it over all states; sample-based methods such as temporal-difference learning apply it to individual observed transitions. The Bellman optimality equation replaces the policy average with a maximum to characterize optimal behavior.