MPI Collective Operations
Collectives are group communications, such as broadcast, reduce, and all-to-all, that libraries implement with efficient algorithms across all ranks.
Communication that involves everyone
A collective operation involves every rank in a communicator, not just a pair. Rather than hand-coding many point-to-point messages, a program calls one collective and the MPI library chooses an efficient algorithm. This is both simpler and faster, because the library exploits the network topology.
The common collectives
- Broadcast: one rank sends the same data to all
- Scatter / Gather: distribute pieces to ranks / collect pieces back
- Reduce / Allreduce: combine values with an operator; Allreduce leaves the result on every rank
- All-to-all: every rank sends distinct data to every other rank
- Barrier: all ranks wait until each has arrived
Why they scale
A naive broadcast from one rank to N others costs N sequential messages, order N time. A tree-based broadcast halves the remaining recipients each step, costing order log N. Reductions use the same tree idea. All-reduce, central to gradient averaging in distributed training, is often done as a reduce-scatter followed by an all-gather, the ring all-reduce, which is bandwidth-optimal.
The synchronization cost
Most collectives are implicit synchronization points: a slow or imbalanced rank delays everyone. This makes collectives a frequent scaling bottleneck and a reason to keep work balanced. Non-blocking collectives allow the group operation to proceed while other work continues.
Where they matter
All-reduce dominates the communication in data-parallel machine learning. Reductions appear in every dot product, norm, and convergence check in iterative solvers. Understanding collective cost is essential to predicting how a distributed code will scale.