Hierarchical Reinforcement Learning
Hierarchical RL decomposes long tasks into reusable sub-behaviors, letting a high-level policy select among temporally extended options.
Decomposing the problem
Hierarchical reinforcement learning (HRL) structures decision making across multiple levels of temporal abstraction. A high-level policy chooses among subgoals or extended behaviors, while low-level policies carry them out step by step. This decomposition tackles long-horizon tasks that flat RL struggles with.
The options framework
The most established formalism is the options framework. An option is a temporally extended action defined by an initiation set (where it can start), an internal policy (how it behaves), and a termination condition (when it ends). The high-level policy selects options rather than primitive actions, and each option runs for many steps before control returns to the top.
Why hierarchy helps
- Shorter effective horizon: the top level makes far fewer decisions, easing credit assignment.
- Reuse: learned sub-behaviors transfer across tasks that share sub-structure.
- Exploration: exploring in the space of subgoals reaches distant states faster than random primitive actions.
- Interpretability: the decomposition often mirrors a human's view of the task.
The challenges
The hard part is discovering good sub-behaviors automatically. Hand-designed subgoals work when the structure is known, but learning useful options from scratch — deciding what the sub-behaviors should be and when they should end — is difficult and remains an active research area. Poorly chosen options can hurt more than they help.
Relation to planning
Hierarchy connects to model-based planning: a high-level model over subgoals can be far smaller and easier to plan with than a step-by-step model. For complex control tasks with natural phases — startup, steady operation, shutdown — a hierarchical decomposition mirrors the operational structure and can make simulated control more tractable.