Hierarchical RL and the Options Framework
Options are temporally extended actions that let an agent plan over skills rather than primitive steps.
Acting at multiple time scales
Hierarchical RL introduces structure over time. Instead of choosing a primitive action each step, a high-level policy selects a skill that runs for many steps before control returns. This shortens the effective horizon, improves exploration, and lets skills be reused across tasks.
The options framework
Sutton, Precup, and Singh formalized a skill as an option, a triple (I, pi, beta): an initiation set I of states where the option can start, an internal policy pi that acts while the option runs, and a termination condition beta(s) giving the probability of ending in each state. A policy over options selects among them; primitive actions are just one-step options.
Semi-Markov decision processes
Because options last variable durations, the decision process over options is a semi-Markov decision process (SMDP). Bellman equations extend naturally: the value of an option accounts for the multi-step discounted reward until termination, then bootstraps from the state where control returns.
- Faster credit assignment: reward propagates across whole skills, not single steps
- Better exploration: options carry the agent through long uninformative regions
- Transfer: a library of options learned on one task speeds learning on the next
Learning the hierarchy
Early work supplied options by hand. Later methods learn them end to end. The option-critic architecture learns option policies and termination functions by gradient descent; feudal and goal-conditioned approaches instead have a manager propose subgoals that a worker must reach. A persistent challenge is preventing degenerate solutions where options terminate every step (collapsing to flat RL) or never terminate (ignoring the hierarchy).