Distributed Computing
Coordinating many networked computers, each with its own memory, to work on a common task.
Definition
Distributed computing spreads a computation across multiple independent computers that communicate over a network, each with its own memory. Unlike shared-memory parallelism, data must be explicitly exchanged between machines.
The CAP principle captures a fundamental tension: under a network partition, a distributed system must sacrifice either consistency or availability. Understanding which guarantee a system provides is essential to reasoning about its behavior when parts of the network fail.
Failure is the defining reality of distributed systems: with many machines, some component is nearly always degraded, so the software must expect and tolerate partial failure rather than assume everything works. Principles such as the CAP trade-off, which forces a choice between consistency and availability under network partitions, formalize the limits. Designing for graceful degradation, retries, and recovery is what separates a robust system from a fragile one.
Challenges
- Communication latency and bandwidth limits.
- Partial failures: some nodes may fail mid-computation.
- Consistency and coordination across nodes.
- Load balancing across heterogeneous machines.
Why it matters
Distributed systems scale beyond the limits of a single machine and underpin cloud computing, large databases, and HPC clusters. Their difficulty lies in coordination and fault tolerance, since more machines mean more ways to fail.
Fusion connection
Large Kronos simulations run distributed across many cluster nodes, using message passing to exchange the plasma-field data that couples neighboring subdomains.