AlphaZero

AlphaZero replaces MCTS rollouts with a learned network and trains purely by self-play from the rules alone.

One network, two heads

AlphaZero uses a single deep network f(s) = (p, v): a policy head p over actions and a scalar value head v estimating the game outcome from state s. This network replaces the random rollouts of classic MCTS. There is no human data and no handcrafted features beyond the raw board.

PUCT search

Search selects actions by a variant of UCT called PUCT: it adds the network's prior p(s,a) as a multiplier on the exploration term, so the network steers where the tree grows. Each simulation descends to a leaf, evaluates it once with v (no rollout), and backpropagates that value. A few hundred simulations per move produce a refined visit-count distribution.

Policy improvement by search

The visit counts from search form an improved policy pi that is stronger than the raw network policy p. This is the key idea: MCTS acts as a policy-improvement operator. Training then regresses the network toward that improved target.

# AlphaZero loss for one position
# targets: pi (search visit distribution), z (game outcome)
# preds:   p  (policy head),            v (value head)
loss = (z - v)**2 - (pi * log(p)).sum() + c * l2_weights

The self-play loop

• Generate games by self-play, using MCTS+network to choose moves
• Store (state, search policy pi, final outcome z) for every position
• Train the network to match pi and z
• Repeat with the improved network

The same algorithm mastered Go, chess, and shogi with no game-specific tuning beyond input and output shapes. AlphaZero shows that search and learning reinforce each other: search generates better targets, and the better network makes each search sharper.