The Policy Gradient Theorem

The policy gradient theorem gives the gradient of expected return with respect to policy parameters without differentiating the environment.

Optimizing the policy directly

Policy-gradient methods parameterize the policy pi(a | s; theta) and improve it by gradient ascent on expected return J(theta). The obstacle is that the return depends on the environment's dynamics, which we cannot differentiate. The policy gradient theorem resolves this.

The result

The theorem states that the gradient of expected return is grad J(theta) = E[ sum over t of grad log pi(a_t | s_t; theta) * Q^pi(s_t, a_t) ]. The gradient of the objective depends only on the gradient of the policy's log-probability and an action-value weight — the unknown transition dynamics drop out entirely.

Why this is remarkable

It converts a hard problem — differentiating through a stochastic, unknown environment — into an expectation the agent can estimate by sampling trajectories. The term grad log pi is the score function; it points in the direction that makes the taken action more likely, and it is scaled by how good that action turned out to be.

The weighting term

The weight on each action can be the return, the action value Q^pi, or, to reduce variance, the advantage A^pi. Subtracting a state-dependent baseline such as V^pi leaves the gradient unbiased while shrinking its variance — the core idea behind actor-critic methods.

What it enables

The theorem is the foundation of a whole family of algorithms: REINFORCE, actor-critic, A2C/A3C, PPO, and more. These methods handle continuous action spaces and stochastic policies naturally, where value-based methods struggle.