Parallel I/O and File Systems
Parallel file systems let thousands of processes read and write shared data at once; I/O is a frequent and overlooked scaling bottleneck.
When storage becomes the limit
Large simulations read big inputs and write big outputs, checkpoints, snapshots, and results. If every process writes its own files or funnels data through one rank, storage becomes a bottleneck that no amount of compute can fix. Parallel I/O lets many processes read and write cooperatively at high aggregate rates.
Parallel file systems
HPC systems use parallel file systems such as Lustre and GPFS that stripe a file across many storage servers, so a large read or write is served by many disks in parallel. Aggregate bandwidth scales with the number of storage targets, but only if the application issues large, aligned, coordinated accesses that spread across them.
I/O strategies
- File per process: simple but creates a metadata storm at large scale
- Single shared file: many processes write one file via MPI-IO
- Collective I/O: processes coordinate to issue large, aligned transfers
High-level libraries
Libraries such as HDF5 and parallel NetCDF sit atop MPI-IO and provide portable, self-describing formats with parallel access. They handle striping, alignment, and metadata, sparing applications the low-level details while achieving good bandwidth. They also make output readable across tools and machines.
The metadata trap
Bandwidth is not the only concern. Creating, opening, and stat-ing huge numbers of files overwhelms the file system's metadata service, a common failure mode of the file-per-process pattern at scale. Writing fewer, larger files and using collective I/O avoids it. Because checkpointing is I/O-heavy, its performance depends directly on getting parallel I/O right.