On-Policy vs Off-Policy Learning
On-policy methods learn about the policy they follow; off-policy methods learn about one policy from data generated by another.
Two ways to use experience
A basic axis in reinforcement learning is whether the policy being learned about (the target policy) is the same as the policy generating the data (the behavior policy). When they coincide the method is on-policy; when they differ it is off-policy.
On-policy
On-policy methods, such as SARSA and standard policy-gradient methods, evaluate and improve the very policy that collects the data, including its exploration. They are more stable and simpler to reason about, but they must discard old data whenever the policy changes, making them less sample efficient.
Off-policy
Off-policy methods, such as Q-learning and DQN, learn a target policy from data produced by a different behavior policy. This lets them reuse stored experience, learn from demonstrations or logged data, and learn the greedy policy while exploring. The price is greater instability and the need to correct for the mismatch between policies.
Correcting the mismatch
Off-policy value methods often correct implicitly through the max operator or bootstrapped targets. Off-policy policy methods use importance sampling, reweighting each transition by the ratio of target to behavior probability. Large ratios cause high variance, which is why techniques like clipping and truncation are common.
Practical implications
- Sample efficiency: off-policy reuses data via replay buffers; on-policy usually cannot.
- Stability: on-policy is generally steadier; off-policy needs stabilizers.
- Data source: off-policy can learn from demonstrations or logs, enabling offline RL.
- Safety: on-policy accounts for the risk of its own exploration.