Checkpoint/Restart and Fault Tolerance
Checkpointing periodically saves a job's state so that after a failure it can restart from the last save rather than from the beginning.
Why long jobs need it
A large parallel job runs on thousands of components, and the more components and the longer the run, the higher the chance one fails before completion. Without protection, a single node failure discards days of computation. Checkpointing periodically writes the application's state to storage; on failure, the job restarts from the most recent checkpoint, so only the work since that point is lost. It also lets a job that exceeds a scheduler time limit continue in a later allocation.
The frequency trade-off
Checkpointing too often wastes time writing state that is rarely needed; too rarely risks losing a large amount of work. The optimal interval balances the cost of writing a checkpoint against the expected work lost per failure, a relationship captured by Young's and Daly's formulas, which set the interval roughly as the square root of twice the checkpoint cost times the mean time between failures. Faster checkpoints (via burst buffers) allow more frequent, cheaper protection.
- Application-level checkpoints save only the state needed to resume.
- Restart resumes from the last checkpoint, not from the start.
- Interval balances write cost against expected work lost.
- Multilevel schemes keep frequent local copies plus rarer durable ones.
Application-level versus system-level
Application-level checkpointing has the program write its own compact state (fields, timestep, RNG state), which is portable and small. System-level checkpointing snapshots the entire process memory transparently but is larger and less portable. Multilevel libraries combine tiers: frequent checkpoints to node-local flash for common single-node failures, and occasional copies to the parallel file system for rarer system-wide failures, getting cheap protection with a durable backstop.
In practice
A multi-day Hyperion run writes application-level checkpoints (mesh fields, step counter, and random-number state) at an interval tuned to the machine's failure rate. After a node failure the campaign resumes from the last checkpoint, and the same mechanism lets a run that hits its wall-time limit continue seamlessly in the next scheduled allocation.