Sample Efficiency in RL
How much interaction an algorithm needs to learn is often the deciding factor when environment steps are slow or costly.
The scarce resource
Sample efficiency measures how much environment interaction an algorithm needs to reach a given level of performance. It matters most when each interaction is expensive — real robots, physical plants, human-in-the-loop systems — as opposed to fast simulators where samples are nearly free and wall-clock time dominates instead.
What drives it
- Data reuse: off-policy methods with a replay buffer reuse each transition many times; on-policy methods often discard data after a few updates.
- Bootstrapping: TD methods learn from partial trajectories, needing fewer complete episodes than Monte Carlo.
- Model use: model-based methods generate synthetic experience, extracting more learning per real step.
- Prior knowledge: demonstrations, curricula, and good state representations reduce the exploration burden.
The trade-off with stability
Sample-efficient methods are often less stable. Off-policy value learning reuses data aggressively but risks divergence from the deadly triad; on-policy methods like PPO are steadier but need more fresh samples. Model-based methods are efficient but vulnerable to model bias. There is rarely a free lunch.
Measuring it fairly
Sample efficiency is reported as a learning curve of performance against environment steps, ideally averaged over many random seeds because RL results vary widely between runs. Comparing algorithms on a single seed is unreliable; robust conclusions require reporting variability across seeds.
Why it matters for control
For controlling costly physical systems, sample efficiency decides whether learning is even feasible. The common answer is to learn primarily in a fast, validated simulator — where samples are cheap — and then transfer the policy, which shifts the challenge from sample efficiency to the sim-to-real gap between the simulator and the true system.