Instruction Pipelining
CPUs overlap the stages of successive instructions in a pipeline, raising throughput; hazards and mispredicted branches stall it.
Overlapping instruction stages
Executing an instruction involves several steps, fetch, decode, execute, access memory, write back. Rather than finishing one instruction before starting the next, a pipelined CPU overlaps them like an assembly line: while one instruction executes, the next decodes and a third is fetched. Ideally the CPU completes one instruction per cycle even though each takes several cycles end to end.
Throughput, not latency
Pipelining does not make a single instruction finish faster; it raises throughput, the rate of completed instructions. This mirrors pipeline parallelism at the software level. Deeper pipelines allow higher clock speeds but pay more when the pipeline must be flushed.
Hazards
- Data hazards: an instruction needs a result not yet produced
- Control hazards: a branch makes the next instruction uncertain
- Structural hazards: two instructions need the same hardware unit
Branch prediction and speculation
A branch stalls the pipeline because the CPU does not yet know which instruction follows. Branch predictors guess the outcome and the CPU speculatively executes down the predicted path. A correct prediction costs nothing; a misprediction flushes the speculative work and restarts, costing many cycles. Predictable branches are therefore much cheaper than random ones.
Superscalar and out-of-order
Modern cores go further: superscalar execution issues several instructions per cycle across multiple pipelines, and out-of-order execution runs independent instructions while a stalled one waits. Together with pipelining these extract instruction-level parallelism, the foundation on which vector, thread, and process parallelism are layered.