REINFORCE

REINFORCE is the simplest policy-gradient algorithm, updating policy parameters using Monte Carlo returns as the action weight.

Monte Carlo policy gradient

REINFORCE is the basic policy-gradient algorithm. It applies the policy gradient theorem using complete-episode returns as the weight on each action. It is model-free, on-policy, and works directly with stochastic policies over discrete or continuous actions.

The update

For each episode, compute the return G_t following each step, then update theta <- theta + alpha * sum over t of grad log pi(a_t | s_t; theta) * G_t. Actions that preceded high returns are made more likely; actions before low returns are made less likely. The gradient is estimated purely from sampled trajectories.

The variance problem

REINFORCE is unbiased but suffers high variance because the full return is noisy and depends on a long chain of random events. This makes learning slow and unstable, and it is the central weakness the method's refinements address.

Baselines

Subtracting a baseline b(s) from the return, giving grad log pi * (G_t - b(s)), reduces variance without adding bias. A natural baseline is the state-value estimate V(s), which turns the weight into an estimate of the advantage. This step leads directly to actor-critic methods, where a learned critic supplies the baseline.

Where it sits

REINFORCE is rarely the best choice in practice, but it is the conceptual root of modern policy optimization. Understanding its update and its variance problem clarifies why later methods — actor-critic, A2C, PPO — add critics, bootstrapping, and trust regions.