Overlapping Communication and Computation
Hiding communication behind useful work is a core scaling technique: start a transfer, compute on independent data, then finish the transfer.
The idle-while-waiting problem
In a distributed code, a process often needs data from a neighbor before it can proceed. If it simply waits for the message, the processor sits idle during the transfer, wasting time that grows with processor count. Overlapping communication with computation hides that latency by doing useful work while the data is in flight.
The pattern
The technique splits work into parts that need remote data and parts that do not. A process starts a non-blocking transfer, then computes on the interior region that depends only on local data, and only when that is done waits for the boundary data to arrive to finish the dependent region. If the interior work takes at least as long as the transfer, the communication is effectively free.
In MPI
- Post non-blocking
MPI_IsendandMPI_Irecvfor halo exchange - Compute the interior cells that need no remote data
- Call
MPI_Wait, then update the boundary cells
On GPUs
The same idea applies to host-device transfers. Using asynchronous copies and streams, a GPU can compute one kernel while copying data for the next, keeping the arithmetic units busy instead of stalling on the slow PCIe or NVLink bus. This is essential for GPU codes that must move data each step.
Why it matters at scale
As machines grow, communication becomes a larger share of run time (strong scaling). Overlap does not reduce the communication, but it removes its cost from the critical path. It is one of the highest-value techniques for making a code scale to large node counts, and a common target of tuning.