The Memory Wall
The memory wall is the widening gap between how fast processors compute and how fast memory can supply data to them.
A growing gap
Over decades, processor arithmetic throughput grew much faster than the bandwidth and latency of main memory improved. The result is the memory wall: for many workloads the processor could compute far faster than memory can feed it, so the arithmetic units sit idle waiting for data. The performance ceiling for such codes is set by memory, not by the compute rate the marketing numbers advertise.
Bandwidth and latency
Two aspects of memory both lag. Bandwidth is how many bytes per second can flow; it limits streaming, throughput-oriented kernels. Latency is how long a single access takes; it limits pointer-chasing and dependent-access patterns where the next request cannot start until the previous returns. High-bandwidth memory (HBM) attacks the first; caches, prefetching, and many concurrent outstanding requests attack the second by overlapping accesses.
- Compute speed outran memory speed, leaving arithmetic units starved.
- Bandwidth limits streaming kernels; latency limits dependent accesses.
- Deep cache hierarchies and HBM exist to soften the wall.
- Locality and reuse are the software response, since faster math will not help.
The software consequence
Because so many scientific kernels (stencils, sparse linear algebra, graph traversal) are memory-bound, most performance work targets memory, not arithmetic. The levers are the ones the roofline and arithmetic-intensity pages describe: reuse data while cached, choose layouts that coalesce and pack well, block computations to fit fast memory, and reduce bytes moved through lower precision or compression. Adding arithmetic capability to a memory-bound code changes nothing.
In practice
A Hyperion sparse solver spends most of its time waiting on scattered memory accesses, not doing arithmetic. Recognizing it as memory-wall-limited redirects tuning toward improving locality and reducing indirection, which move the wall, rather than toward faster floating-point paths that the wall makes irrelevant.