Synchronization Primitives
Synchronization coordinates concurrent execution through locks, atomics, and barriers, trading some serialization for correctness.
Coordinating concurrency
When threads or processes share state or must proceed in a certain order, they need synchronization: mechanisms that constrain the otherwise free interleaving of concurrent execution. Synchronization prevents races and enforces ordering, at the cost of some serialization and overhead.
The primitives
- Mutex (lock): only one thread holds it at a time; guards a critical section
- Atomic operation: an indivisible read-modify-write, no lock needed
- Barrier: all participants wait until every one has arrived, then proceed
- Condition variable: a thread sleeps until another signals a condition
- Semaphore: permits up to a fixed number of concurrent holders
Locks and their hazards
Locks are simple but dangerous. Holding a lock too long serializes the program and throttles scaling. Acquiring multiple locks in inconsistent orders causes deadlock, where each thread waits forever for a lock another holds. Consistent lock ordering, minimal critical sections, and fine-grained locking mitigate these hazards.
Barriers in HPC
Barriers are ubiquitous in bulk-synchronous parallel codes: compute a step, synchronize, exchange data, repeat. A barrier is only as fast as the slowest participant, so it is where load imbalance shows up as wasted time. Reducing the frequency and cost of global synchronization is a recurring scaling optimization.
Lock-free alternatives
Atomics enable lock-free structures that avoid the blocking and deadlock risk of locks, though they are subtle to implement correctly. For many HPC patterns the cleanest approach is to avoid shared mutable state entirely, using per-thread data and combining results with reductions, sidestepping most synchronization.