Policy Gradient
A reinforcement learning approach that optimizes a parameterized policy directly by gradient ascent on expected reward.
Definition
Policy gradient methods represent the policy as a parameterized, usually probabilistic, function and adjust its parameters directly to increase expected reward, following the gradient of return with respect to the policy.
The high variance of the gradient estimate is the central practical problem, addressed by subtracting a baseline and by estimating an advantage function that measures how much better an action is than average. These variance-reduction ideas define the actor-critic family.
Proximal policy optimization has become a workhorse because it limits how far each update moves the policy, avoiding the collapses that plague naive policy gradients. The same family of methods is used to align large language models to human preferences, where a learned reward model supplies the signal. This breadth, from robot control to model alignment, reflects how general the policy-optimization framing is.
Because the policy is stochastic and differentiable, it handles continuous action spaces naturally, unlike value-based methods that must search over actions.
Refinements
- Baselines and advantage estimates reduce gradient variance.
- Actor-critic methods pair a policy with a learned value estimate.
- Trust-region and clipped objectives (PPO) stabilize updates.
Why it matters
Policy gradients power much of modern reinforcement learning for robotics and continuous control, and related methods are used to align large language models. Their challenge is high variance and sample inefficiency.
Fusion connection
Continuous control of coil currents to shape a plasma is a natural policy-gradient problem, explored in simulation for tokamak control ahead of any hardware.