MPI Allreduce
Allreduce combines a value from every process using a reduction operator and delivers the identical result back to all processes.
The operation
MPI_Allreduce takes one buffer per process, applies an associative reduction operator (sum, max, min, product, logical-and, or a user-defined operator) across all ranks, and returns the combined result to every rank. It is logically a reduce to one root followed by a broadcast, but implementations fuse the two so the cost is a single collective rather than two.
Why it dominates
Allreduce is one of the most frequently called collectives in scientific and machine-learning workloads. Global convergence checks, dot products in Krylov solvers (conjugate gradient, GMRES), normalization, and gradient averaging in data-parallel training are all allreduces. Because it appears inside iterative loops, its latency is often on the critical path, so its performance shapes overall scaling.
Algorithms
For small messages, recursive doubling gives a latency of about log2(P) message steps: in each step, ranks pair up across a growing stride and exchange partial results. For large messages, ring allreduce is bandwidth-optimal: data is split into P chunks that circulate around a logical ring in a reduce-scatter phase followed by an allgather phase, so each link carries roughly 2(P-1)/P of the buffer. Modern libraries switch between these by message size.
from mpi4py import MPI
import numpy as np
comm = MPI.COMM_WORLD
local = np.array([comm.Get_rank() + 1.0])
total = np.zeros(1)
comm.Allreduce(local, total, op=MPI.SUM)
# every rank now holds the same sum 1+2+...+PNumerical caution
Floating-point addition is not associative, so the order in which partial sums are combined can change the last bits of the result. Different process counts, or different internal algorithms, can produce slightly different sums. Reproducible allreduce implementations fix the reduction order or use compensated summation when bitwise determinism matters for validation.
In a Hyperion breeder solver, the residual norm checked each iteration is an allreduce; making it reproducible keeps convergence traces comparable across runs on different node counts.