Heterogeneous Computing

Heterogeneous systems combine CPUs with accelerators, assigning each part of a workload to the processor that runs it best.

Different engines for different work

A heterogeneous node pairs general-purpose CPUs with accelerators, usually GPUs. The CPU handles control flow, irregular code, and orchestration; the accelerator handles the arithmetic-heavy, data-parallel inner loops. Nearly all leading HPC systems are heterogeneous because no single processor is best at everything.

The programmer's burden

Heterogeneity complicates software. The two processor types have separate memory spaces (or a shared space with non-uniform performance), so data must be placed and moved deliberately. Work must be partitioned to keep both busy, and the fast accelerator must not stall waiting on the slower CPU or on data transfers.

Managing data movement

• Keep data resident on the accelerator across many kernels
• Overlap host-device transfers with computation
• Use unified or managed memory carefully; convenience can hide costly migrations

Portability approaches

Writing separate code per device is unsustainable, so portable models have emerged: OpenMP target offload, SYCL, Kokkos, RAJA, and HIP let one source target multiple vendors' accelerators. They trade some peak performance for the ability to run across machines, an increasingly important consideration as accelerator vendors diversify.

Why it dominates

Accelerators deliver far more arithmetic per watt, and power is the binding constraint at large scale. So heterogeneity is not a stylistic choice but a consequence of energy limits. The design lesson is consistent: expose massive data parallelism, minimize data movement, and keep the accelerators fed.