Parallel I/O
Parallel I/O lets many processes read and write a shared file concurrently through coordinated interfaces such as MPI-IO and parallel HDF5.
The bottleneck
When thousands of processes each write their own file, the file system is flooded with metadata operations and small writes, and the result is a directory of thousands of fragments that are painful to manage and analyze. When one process gathers all data and writes serially, that single stream cannot use the file system's aggregate bandwidth. Parallel I/O is the middle path: many processes cooperatively read and write a single shared file.
MPI-IO
MPI-IO, part of the MPI standard, is the foundation. Processes open a file collectively and each writes to its own region, described by an MPI file view built from derived datatypes. Its most important feature is collective I/O (the collective read/write calls), which lets the library aggregate many small, scattered process writes into a few large, contiguous transfers to the storage system, a transformation called two-phase I/O that dramatically improves throughput.
- One shared file avoids the file-per-process metadata storm.
- Collective calls aggregate small writes into large efficient transfers.
- File views map each process to its region of the shared file.
- Higher-level libraries add portable, self-describing formats on top.
Self-describing formats
Directly managing byte offsets is error-prone, so most applications use higher-level libraries built on MPI-IO. Parallel HDF5 and parallel NetCDF store data in portable, self-describing files that record dimensions, types, and metadata alongside the values. They handle the parallel layout while giving the application a clean model of named datasets, so files are readable on any machine and by standard analysis tools.
In practice
A Hyperion simulation writes its distributed field as one parallel HDF5 dataset, each rank contributing its subdomain through collective I/O. The output is a single portable file that analysis tools open directly, and the collective aggregation lets the write use the parallel file system's full bandwidth instead of a single serial stream.