The Memory Hierarchy
Memory is organized as a hierarchy from fast, small registers to slow, large storage, and performance depends on keeping data close to the processor.
Fast is small, big is slow
No memory technology is simultaneously fast, large, and affordable, so computers layer several. Each level up the hierarchy is faster and smaller; each level down is larger and slower. The processor reads from the fastest level that holds the data, falling back to slower levels on a miss.
The levels
- Registers: a few hundred bytes, sub-nanosecond access
- L1 / L2 / L3 caches: kilobytes to tens of megabytes, single-digit to tens of nanoseconds
- Main memory (DRAM): gigabytes to terabytes, around a hundred nanoseconds
- Storage (SSD, disk): terabytes and up, microseconds to milliseconds
Latency in perspective
If a register access were one second, an L1 hit would be a few seconds, a main-memory access a couple of minutes, and a disk read many days. These ratios explain why data placement, not raw arithmetic rate, so often decides real performance.
Locality is the lever
The hierarchy pays off because programs exhibit locality. Temporal locality: recently used data is likely used again. Spatial locality: data near recently used data is likely used soon. Algorithms that reuse data while it sits in cache, and stream memory contiguously, run far faster than those that jump around.
The bandwidth wall
Arithmetic throughput has grown faster than memory bandwidth for decades, so many codes are memory-bound: limited by how fast data arrives, not how fast it can be computed on. Blocking (tiling) loops to fit working sets in cache is the standard remedy.