Reward Shaping
Reward shaping adds guidance signals to sparse rewards to speed learning, but poorly designed shaping can change what the agent learns.
Guiding the learner
Reward shaping modifies the reward signal to make learning faster or more reliable, usually by adding intermediate rewards that guide the agent toward good behavior when the true reward is sparse. Done well it accelerates learning; done carelessly it teaches the agent to pursue the shaped signal instead of the real goal.
The sparse-reward problem
Many natural rewards are sparse: a game is won or lost only at the end, a control target is either held or not. With rare feedback the agent may wander for a long time before stumbling on any reward, and credit assignment across a long gap is hard. Shaping supplies a denser gradient to climb.
Reward hacking
The danger is reward hacking: the agent finds a way to score high on the shaped reward without achieving the intended goal. A racing agent rewarded for speed may spin in circles through a checkpoint; an agent rewarded for proximity may hover near a goal it never reaches. The agent optimizes exactly what you wrote, not what you meant.
Potential-based shaping
There is a safe form. Potential-based reward shaping adds a term gamma * Phi(s') - Phi(s), the difference of a potential function over states. This provably leaves the set of optimal policies unchanged while still guiding learning, because the added terms telescope over any trajectory and cancel in the long-run return. It is the principled way to shape.
Practical guidance
- Prefer potential-based shaping when you can define a sensible potential.
- Keep the true objective as the dominant reward; shaping should nudge, not dominate.
- Test whether high shaped reward actually corresponds to task success.
- Watch for degenerate behaviors that exploit the shaped term.