Communication-Avoiding Algorithms
Communication-avoiding algorithms are redesigned to move less data, accepting extra arithmetic because communication, not computation, is the true cost.
Communication is the cost
On modern machines, moving data, between cache and memory, between nodes, is far slower and more energy-hungry than performing arithmetic. Classic algorithms were designed to minimize operations, the resource that was once scarce. Communication-avoiding algorithms invert this: they are redesigned to minimize data movement, even at the price of doing extra arithmetic or storing more, because communication now dominates both time and energy.
Lower bounds and blocking
Theory establishes lower bounds on how little communication an algorithm can perform for a given problem, and communication-avoiding algorithms aim to meet those bounds. The basic tool is blocking: restructuring computation so a block of data brought into fast memory is reused for as much work as possible before being evicted, which is the same idea as cache blocking and directly raises arithmetic intensity. Matrix multiplication and dense factorizations have well-known communication-optimal blocked forms.
- Minimize data moved, not operations performed.
- Blocking reuses fetched data before eviction, meeting communication bounds.
- Some variants recompute values to avoid fetching or exchanging them.
- Trades extra arithmetic or memory for far less communication.
Communication-avoiding Krylov methods
Iterative solvers such as conjugate gradient and GMRES normally do a global reduction (a dot product allreduce) every iteration, and those reductions become the scaling bottleneck at large process counts. Communication-avoiding Krylov methods restructure the algorithm to take several steps between reductions, batching the global communication. This cuts the number of latency-bound collectives, at the cost of extra local work and some numerical care to preserve stability.
In practice
A Hyperion field solver based on a Krylov method can adopt a communication-avoiding variant so its inner loop performs fewer global allreduces per unit of progress. On thousands of ranks, where the allreduce latency was the wall, this restructuring restores scaling that reducing arithmetic never could.