Experience Replay
Experience replay stores past transitions and reuses them in random minibatches, breaking correlations and improving sample efficiency.
Reusing the past
Experience replay is a technique in off-policy reinforcement learning where the agent stores transitions (state, action, reward, next state) in a buffer and later trains on random samples from it, rather than only on the most recent transition. It was a key ingredient in stabilizing the deep Q-network.
Why it helps
- Decorrelation: consecutive transitions are highly correlated; random sampling yields more independent training batches, which gradient methods prefer.
- Sample efficiency: each expensive interaction is reused many times instead of once.
- Stability: mixing old and new data smooths the training distribution and reduces catastrophic forgetting.
Prioritized replay
Uniform sampling treats every transition equally, but some carry more information. Prioritized experience replay samples transitions in proportion to their TD error, focusing learning on surprising or poorly predicted events. To stay unbiased it corrects the skewed sampling with importance-sampling weights.
Buffer design
The buffer is typically a fixed-size ring that discards the oldest transitions once full. Its capacity trades off diversity against staleness: too small and correlations return, too large and very old, off-distribution data can slow adaptation. Hindsight experience replay extends the idea by relabeling failed episodes with achieved goals to extract learning signal from sparse rewards.
When it applies
Replay requires off-policy learning, since stored transitions were generated by older policies. On-policy methods such as standard PPO cannot use a long-lived replay buffer and instead reuse a fresh batch a few times before discarding it. Replay is also the mechanism that makes offline RL conceivable, learning entirely from a fixed logged dataset.