Lustre Striping
Striping controls how many storage targets a Lustre file is spread across and in what chunk size, tuning single-file bandwidth.
Spreading a file
In Lustre, a file's contents are divided into chunks and distributed across one or more Object Storage Targets (OSTs). Two parameters govern this: the stripe count (how many OSTs the file uses) and the stripe size (how large each chunk is before moving to the next OST). Together they set how much parallel bandwidth a single file can draw and how the load spreads across storage servers.
Choosing the parameters
A small file written by one process gains nothing from wide striping and may just add overhead, so a stripe count of one is fine. A large file written by many processes benefits from a high stripe count so that concurrent writes hit different OSTs in parallel rather than contending on one. The stripe size should align with the application's write sizes so that a process's contiguous writes map cleanly onto stripes rather than straddling boundaries.
- Stripe count: number of OSTs the file is spread across.
- Stripe size: bytes written to one OST before advancing to the next.
- Big shared files: high stripe count for parallel bandwidth.
- Small files: single stripe to avoid needless overhead.
How it is set
Striping is set per file or per directory before data is written, since it governs the initial layout. Command-line tools and library hints (including through MPI-IO and parallel HDF5) let an application request a layout. A common mistake is inheriting a default single-stripe layout for a large collective output, which serializes the write onto one OST and wastes the file system's parallelism.
In practice
A large Hyperion field checkpoint written collectively is placed in a directory configured with a stripe count matched to the number of writers and a stripe size aligned to each rank's contribution. This lets the checkpoint write and later read draw the file system's full parallel bandwidth instead of bottlenecking on a single storage target.