Checkpointing and Restart
Checkpointing periodically saves a computation's state so a long run can resume after a failure instead of starting over.
Why long runs need it
A simulation may run for days across thousands of components. The more components and the longer the run, the higher the chance that something fails before completion. Checkpointing writes the full application state to storage at intervals; if a failure occurs, the job restarts from the last checkpoint rather than from the beginning.
The core trade-off
Checkpointing costs time and storage bandwidth, but reduces the work lost to a failure. Checkpoint too often and I/O overhead dominates; too rarely and a failure wastes hours of compute. The optimal interval balances checkpoint cost against expected failure rate, captured by Young's and Daly's formulas, which set the interval near the square root of twice the checkpoint time times the mean time between failures.
Techniques
- Coordinated: all processes pause and save a globally consistent state
- Multilevel: fast local (node memory or SSD) checkpoints backed by slower reliable storage
- Incremental: save only what changed since the last checkpoint
- Asynchronous: overlap writing the checkpoint with continued computation
The I/O bottleneck
A full checkpoint of a large simulation can be enormous, and writing it stresses the parallel file system. This is why multilevel schemes stage checkpoints to fast node-local storage first and flush to reliable storage in the background, and why compression and selective state-saving matter.
Beyond failure recovery
Checkpoints double as scientific output and as a way to split a campaign across scheduler time limits: a job that exceeds its allotted window can stop at a checkpoint and resume in a later allocation, a routine pattern in shared HPC facilities.