Fault Tolerance at Scale
With millions of components, failures are routine; fault tolerance keeps long computations progressing despite hardware and software faults.
Failure is the normal case
A large HPC system has millions of components: processors, memory modules, network links, disks. Even with highly reliable parts, the sheer count means the mean time between failures of the whole system can be hours, not years. At scale, a long run will almost certainly encounter a failure, so codes must expect and survive them.
Kinds of failure
- Fail-stop: a node crashes and stops responding
- Silent data corruption: a bit flips and computation continues with a wrong value
- Transient: a temporary fault that clears on retry
- Network: a link or switch fails, partitioning communication
The main defense: checkpoint and restart
The workhorse of fault tolerance is checkpointing: save state periodically, and after a failure restart from the last good checkpoint. Multilevel checkpointing stages saves to fast local storage and slower reliable storage, balancing checkpoint cost against the work at risk. The optimal interval scales with the square root of the checkpoint cost times the mean time between failures.
Detecting silent corruption
Fail-stop faults are obvious; silent data corruption is insidious because the run continues with a wrong answer. Hardware ECC catches many memory bit flips; algorithm-based fault tolerance adds checksums to linear-algebra operations that detect and sometimes correct errors; and redundant computation can catch the rest, at a cost.
Beyond checkpointing
Some frameworks support fault-tolerant MPI that lets a job continue after losing ranks, and task-based runtimes can re-execute failed tasks. But for most production science, disciplined checkpointing plus hardware ECC remains the practical foundation of resilience, which is why exascale planning treated fault tolerance as a first-order design constraint.