Return and Discounting
The return is the cumulative reward an agent seeks to maximize; the discount factor sets how much future reward counts against the present.
What the agent maximizes
The agent does not maximize immediate reward but the return, the total reward accumulated from a time step onward. Writing rewards as R_{t+1}, R_{t+2}, ..., the discounted return is G_t = R_{t+1} + gamma R_{t+2} + gamma^2 R_{t+3} + ... = sum over k of gamma^k R_{t+k+1}.
The discount factor gamma
The discount factor gamma, between 0 and 1, weights how strongly future rewards count. With gamma near 0 the agent is myopic, caring almost only about the next reward. With gamma near 1 it is far-sighted, valuing distant rewards nearly as much as immediate ones. Discounting also keeps the return finite in continuing tasks where reward accrues forever.
Why discount
- Mathematical: guarantees a finite sum in infinite-horizon problems.
- Modeling: captures genuine preference for sooner rewards.
- Uncertainty: distant futures are less predictable, so weighting them less is reasonable.
- Stability: it reduces variance in value estimates and stabilizes learning.
Episodic returns
In episodic tasks the sum runs only to the terminal step T, so undiscounted returns (gamma = 1) are well defined. Many episodic algorithms still discount slightly to reduce variance and encourage reaching goals sooner.
Effective horizon
A useful rule of thumb: 1/(1 - gamma) approximates the number of steps the agent effectively plans over. A discount of 0.99 gives an effective horizon around 100 steps. Choosing gamma is therefore a modeling decision about how far ahead the task truly requires the agent to look.