Parallel Computing
Performing many computations simultaneously by dividing work across multiple processors.
Definition
Parallel computing splits a computation into parts that execute at the same time on multiple processing units. It exploits the fact that many problems contain independent work that need not be done in sequence.
Correctness is the hidden difficulty: race conditions and deadlocks arise when parallel tasks share state without proper coordination, and such bugs are hard to reproduce because they depend on timing. Disciplined use of synchronization and immutable data reduces this risk.
The hardest part is often correctness rather than speed: shared state accessed without proper coordination produces race conditions and deadlocks that appear only under particular timings and are notoriously difficult to reproduce. Disciplined approaches, immutable data, message passing, and well-scoped synchronization, contain this complexity. Getting parallel code both fast and correct is a distinct skill, and premature optimization here often introduces subtle bugs.
Forms of parallelism
- Data parallelism: the same operation on many data elements.
- Task parallelism: different operations at once.
- Shared-memory (threads) vs distributed-memory (message passing).
Limits
Speedup is bounded by the fraction of work that must remain sequential, as Amdahl's law quantifies. Communication and synchronization between processors add overhead that can erode gains if not managed.
Why it matters
With single-core clock speeds plateaued, parallelism is the main route to more performance. Writing correct, efficient parallel code, avoiding races and balancing load, is a central skill in scientific computing.
Fusion connection
Plasma simulations parallelize by dividing the physical domain across processors, so that each handles part of the machine while exchanging boundary data with its neighbors.