Multi-Agent Reinforcement Learning
When several learning agents share an environment, each faces a moving target as the others adapt, reshaping the learning problem.
Many agents, one world
Multi-agent reinforcement learning (MARL) studies settings where several agents learn and act in a shared environment. Each agent's reward and next state depend not only on its own actions but on the actions of the others, which are themselves changing as they learn. This coupling makes MARL fundamentally harder than single-agent RL.
Non-stationarity
From any one agent's viewpoint, the environment is non-stationary: as the other agents update their policies, the effective dynamics and rewards shift. This violates the stationary-MDP assumption that most single-agent guarantees rely on, so convergence is no longer assured and learning can oscillate or chase a moving target.
Cooperation, competition, mixed
- Cooperative: agents share a goal and must coordinate, as in a team.
- Competitive: agents have opposing goals, as in two-player games, where equilibrium concepts from game theory apply.
- Mixed: elements of both, as in most realistic multi-party settings.
Common approaches
Centralized training with decentralized execution is a leading paradigm: during training a critic can see all agents' observations and actions, reducing non-stationarity, while at execution each agent acts on only its own local observation. Other techniques include explicit communication channels, opponent modeling, and equilibrium-seeking algorithms grounded in game theory.
Emergent behavior
MARL can produce rich emergent phenomena — coordination, division of labor, competition, even tacit communication — that no single agent was programmed to exhibit. This makes it a powerful tool for studying strategic interaction, but also a source of unpredictability that must be managed carefully in any system where the agents' collective behavior has real consequences.