Meta-Reinforcement Learning
Meta-RL learns how to learn, producing agents that adapt to a new task within a handful of episodes.
Learning to learn
Standard RL learns one task from scratch. Meta-RL trains across a distribution of related tasks so that the resulting agent can adapt rapidly to a new task drawn from that distribution. The objective is not performance on any single task but fast adaptation across many.
Two main families
Optimization-based meta-RL, exemplified by MAML, learns an initial set of parameters from which a few gradient steps on a new task reach good performance. Context-based (recurrent) meta-RL, such as RL^2, treats adaptation as inference: a recurrent policy consumes rewards and observations across an episode, and its hidden state encodes what has been learned about the current task, so adaptation happens in the network's activations with no weight updates at test time.
# MAML meta-objective (schematic)
# for each task: theta' = theta - alpha * grad(task_loss(theta))
# meta-update: theta = theta - beta * grad( sum over tasks
# of task_loss(theta') )Adaptation as a POMDP
Fast adaptation is really inference about which task the agent faces, an unknown parameter of a POMDP. Context-based methods like PEARL make this explicit: they infer a probabilistic task variable from recent experience and condition the policy on it, combining posterior sampling for exploration with off-policy efficiency.
Why it matters and where it strains
- Sample-efficient adaptation to new but related tasks, useful for robotics and control
- Generalization is bounded by the training task distribution; far-off tasks fail
- Meta-training itself is costly and can be unstable
Meta-RL formalizes transfer and few-shot adaptation for sequential decisions. Its central lesson, that the exploration strategy for a new task can itself be learned, connects it to curiosity, POMDPs, and Bayesian RL.