Parallelism and Concurrency
Concurrency is structuring work to overlap; parallelism is running it truly at the same time. Both speed up modern computing.
Two related ideas
Concurrency is about dealing with many things at once — structuring a program so tasks can make progress independently. Parallelism is about doing many things at once — actually executing on multiple processors simultaneously. A program can be concurrent without being parallel, and vice versa.
Amdahl’s law
The speedup from parallelism is capped by the fraction of work that must run sequentially. If one tenth of a program is inherently serial, no amount of hardware makes it more than ten times faster. This law tempers expectations and directs effort toward the serial bottlenecks.
The hazards
- Race conditions: results depend on unpredictable ordering of operations.
- Deadlocks: tasks wait on each other forever.
- Nondeterminism: parallel reductions can change the last digit of a sum.
Determinism under parallelism
Because floating-point addition is not associative, summing numbers in a different order can yield a slightly different result. Parallel programs that split and recombine sums may therefore give run-to-run variation. Where reproducibility matters, deterministic reduction orders are used deliberately.
Getting the model right
Choosing the correct concurrency model — shared memory, message passing, or task-based — depends on the problem and the hardware. The wrong model can make code slower and buggier than a clean serial version, so parallelism is applied where it demonstrably pays, not as a reflex.