Policies

A policy is the agent's behavior: a rule, deterministic or stochastic, that maps states to actions.

The agent's behavior

A policy, written pi, defines how the agent acts. A deterministic policy pi(s) returns a single action for each state. A stochastic policy pi(a | s) gives a probability distribution over actions given the state. The policy is the object reinforcement learning ultimately wants to optimize.

Deterministic versus stochastic

Deterministic policies are simple and often optimal in fully observable MDPs with known dynamics. Stochastic policies are essential when exploration must be built into behavior, when the environment is partially observable, or in adversarial and multi-agent settings where being predictable is a weakness. Policy-gradient methods naturally learn stochastic policies.

Representing policies

• Tabular: one entry per state, feasible only for small discrete problems.
• Linear: a weighted combination of state features.
• Neural: a network mapping state to action or to action probabilities, needed for large or continuous spaces.

Optimal policies

An optimal policy pi* achieves the highest expected return from every state. Every finite MDP has at least one optimal policy, and there is always a deterministic optimal policy. Optimal policies share the same optimal value function, which the Bellman optimality equation characterizes.

Finding good policies

Two broad families search for good policies. Value-based methods learn a value function and derive the policy by acting greedily with respect to it. Policy-based methods parameterize the policy directly and improve it by gradient ascent on expected return. Actor-critic methods combine both.