Memory Hierarchy

The memory hierarchy stacks storage from small and fast to large and slow, using locality to give the illusion of both.

The problem

Fast memory is small and large memory is slow; no single technology is both. The memory hierarchy resolves this by layering storage so that the fast levels hold the data currently in use and the slow levels hold everything else.

The levels

From fastest to largest: processor registers, then one or more levels of SRAM cache, then DRAM main memory, then non-volatile storage. Each level is larger and slower than the one above and acts as a backing store for it.

Locality

The hierarchy works because programs exhibit locality. Temporal locality means recently used data is likely to be used again soon; spatial locality means data near recently used data is likely to be used next. Caches exploit both by keeping recent items and their neighbors close.

Caches and blocks

A cache holds copies of data from the level below, organized in blocks. A hit serves data from the fast level; a miss fetches a block from the slower level. The average access time is close to the fast level's speed when hit rates are high.

The payoff

With good locality and hit rates, the hierarchy delivers nearly the speed of the fastest level at nearly the capacity of the largest, which is why every modern processor is built around it.