Generalized Policy Iteration
Nearly every RL algorithm is a case of two processes — value estimation and policy improvement — pushing toward mutual consistency.
A unifying idea
Generalized policy iteration (GPI) is the observation that almost all reinforcement learning methods can be seen as two interacting processes: one that makes the value function consistent with the current policy, and one that makes the policy greedy with respect to the current value function. They pull toward each other until they agree.
The two forces
- Evaluation: drive the value estimate toward the true value of the current policy.
- Improvement: change the policy to be greedy with respect to the current value estimate.
Evaluation makes the value function correct for the policy; improvement makes the policy correct for the value function. Each step can undo the other's consistency, but over time they converge on a joint fixed point where neither can improve — the optimal policy and its value function.
Not just full sweeps
GPI does not require completing either process before starting the other. Value iteration improves after a single evaluation sweep. Q-learning updates value estimates from single transitions while acting near-greedily. Actor-critic methods run a critic (evaluation) and an actor (improvement) in parallel.
Why the framing helps
Viewing an algorithm through GPI clarifies what could go wrong. If evaluation lags far behind a rapidly changing policy, estimates become stale and learning destabilizes. If improvement is too aggressive relative to noisy value estimates, the policy chases noise. Many practical tricks — target networks, trust regions, slow policy updates — are ways to keep the two processes in a healthy balance.
The fixed point
The joint process settles only when the policy is greedy with respect to its own value function, which is exactly the Bellman optimality condition. GPI is thus the algorithmic shadow of that equation.