GPUDirect and GPU-Aware MPI
GPU-aware communication moves data directly between GPUs across nodes without a detour through host memory, cutting latency and freeing the CPU.
The staging problem
Without GPU-aware support, sending data from one node's GPU to another's requires three steps: copy from GPU to host memory, send host to host over the network, copy from host to the remote GPU. The two extra copies add latency and consume PCIe bandwidth and CPU time. GPUDirect technologies remove them by letting the network adapter access GPU memory directly.
What GPU-aware MPI does
A GPU-aware MPI implementation accepts device pointers in its send and receive calls. Under the hood it uses GPUDirect RDMA so the network interface reads directly from GPU memory and writes directly into the remote GPU memory. For transfers between GPUs on the same node, GPUDirect Peer-to-Peer moves data over NVLink or PCIe without touching host memory at all. The application code looks the same as ordinary MPI but passes GPU buffers.
- GPUDirect RDMA: network adapter reads/writes GPU memory directly.
- GPUDirect P2P: intra-node GPU-to-GPU transfers bypass host memory.
- GPU-aware MPI accepts device pointers in send/receive calls.
- Removes two staging copies, lowering latency and CPU load.
When the benefit is largest
The gain is greatest for latency-sensitive, GPU-resident workloads that communicate frequently, such as domain-decomposed solvers whose halo exchanges happen every timestep between GPUs. Combined with streams, the transfers can also overlap with kernels. The benefit is smaller when data must be reorganized on the host anyway or when messages are so large that the staging copies are a minor fraction of total time.
In practice
A Hyperion field solver that keeps its mesh on GPUs exchanges subdomain halos with GPU-aware MPI, so boundary data hops GPU-to-GPU across the InfiniBand fabric without host staging. This keeps the physics on the device and the CPU free for orchestration and I/O.