Pipeline Parallelism

Pipeline parallelism streams data through a chain of stages, each running on its own processor, so all stages work concurrently on different items.

An assembly line for computation

In a pipeline, a computation is split into ordered stages. Item one enters stage one; when it moves to stage two, item two enters stage one, and so on. Once the pipeline is full, every stage is busy on a different item at the same time. Throughput rises to roughly one completed item per stage time, even though each individual item still traverses the whole pipeline.

Fill, drain, and throughput

A pipeline with k stages needs k-1 steps to fill before all stages are active, and k-1 steps to drain at the end. For a long stream this overhead is negligible; for a short one it dominates. Sustained throughput is set by the slowest stage, so balancing stage durations matters as much as adding stages.

Examples

• CPU instruction pipelines: fetch, decode, execute, memory, write-back
• Data ingestion: read, decode, transform, write
• Deep-learning pipeline parallelism: successive layers placed on successive devices

Pipeline parallelism in model training

When a neural network is too large for one accelerator, its layers are partitioned across devices. A micro-batch flows forward through the devices, then gradients flow back. Naively this leaves devices idle (the pipeline bubble); schemes that interleave many micro-batches shrink the bubble and raise utilization.

Costs

Pipelines introduce buffering between stages and are sensitive to imbalance and stalls. If one stage occasionally blocks, the whole line backs up. Good designs keep stage times uniform, buffer generously, and overlap the pipeline with other forms of parallelism.