MPI Reduce

Reduce combines a value from every process using an operator and delivers the single result to one root process.

The operation

MPI_Reduce applies an associative operator across contributions from all ranks and leaves the result only on a designated root. It is allreduce without the final broadcast, so it costs roughly half the traffic when only one process needs the answer, for instance the rank that will write output or make a control decision.

Built-in and custom operators

MPI provides sum, product, min, max, logical and bitwise and/or/xor, plus the location-aware MPI_MINLOC and MPI_MAXLOC that return both the extreme value and the rank that held it. Applications can register a custom operator with MPI_Op_create; the operator must be associative, and the library assumes it is commutative unless told otherwise, since that assumption enables more parallel schedules.

• Result lands on the root only, unlike allreduce.
• MINLOC/MAXLOC return the value and its owning rank together.
• Custom operators must be associative to be correct under reordering.
• Reduce-scatter is a fused reduce plus scatter used inside ring allreduce.

Tree schedule

Reduce uses a binomial tree in reverse relative to broadcast: leaves send partial results up toward the root, each internal node combining its children's contributions with its own. This finishes in ceil(log2 P) steps. Because floating-point sums are order-dependent, the tree shape can affect the last bits of the result, the same reproducibility concern that applies to allreduce.

from mpi4py import MPI
import numpy as np
comm = MPI.COMM_WORLD
local_max = np.array([float(comm.Get_rank())**2])
global_max = np.zeros(1)
comm.Reduce(local_max, global_max, op=MPI.MAX, root=0)
if comm.Get_rank() == 0:
    print('largest square:', global_max[0])

When to prefer it

Use reduce, not allreduce, when only one rank consumes the result, such as computing the peak neutron flux in a Hyperion run for a log line written by rank 0. Using allreduce there wastes the broadcast half of the traffic on ranks that discard the value.