Multi-Agent Reinforcement Learning
When many learning agents share an environment, each faces a moving target: the others are learning too.
Beyond a single agent
Multi-agent RL (MARL) studies several agents learning simultaneously in a shared environment. It generalizes the MDP to a stochastic (Markov) game, where the transition and each agent's reward depend on the joint action of all agents. Settings range from fully cooperative (shared reward) to fully competitive (zero-sum) to mixed.
Non-stationarity
The core difficulty is that from any one agent's view the environment is non-stationary: as other agents update their policies, the effective dynamics change, so past experience becomes stale and naive independent learning can oscillate or diverge. Convergence guarantees from single-agent RL no longer hold.
Centralized training, decentralized execution
A dominant paradigm is CTDE: during training a critic may see the global state and all agents' actions, which stabilizes learning; at execution each agent acts only on its own local observation. MADDPG and value-decomposition methods (VDN, QMIX) follow this pattern, the latter factoring a joint value into per-agent components while preserving a consistency condition.
- Cooperative: agents share a reward and must coordinate
- Competitive: one agent's gain is another's loss; self-play applies
- Mixed / general-sum: both cooperation and competition, as in team games
Equilibria and emergent behavior
Solution concepts borrow from game theory: a Nash equilibrium is a joint policy where no agent can improve by deviating alone. MARL research shows striking emergent phenomena, including tool use, division of labor, and communication protocols, arising purely from reward. Credit assignment across agents and scalable coordination remain the field's hardest open problems.