Data Systems

HDF5: Hierarchical Scientific Data

HDF5 is a self-describing container format for large, multidimensional numeric arrays organized in a directory-like hierarchy.

A filesystem inside a file

HDF5 (Hierarchical Data Format version 5) stores data as a tree of groups and datasets inside a single file. Groups act like folders; datasets are typed, multidimensional arrays. Any node can carry attributes, small pieces of metadata attached directly to the data they describe. The format is self-describing: the file records its own structure and types, so a reader needs no external schema.

Core objects

Kronos motion — data assimilation

Group: a named container that holds datasets and other groups.
Dataset: an n-dimensional array with a fixed element type and a shape.
Attribute: named metadata attached to a group or dataset.
Dataspace and datatype: descriptions of an array's shape and element format.

Chunking and compression

Large datasets are stored in chunks: fixed-size blocks that can be read and compressed independently. Chunking enables partial reads of a huge array and per-chunk compression such as gzip or szip. Choosing a chunk shape that matches how the data is queried is one of the most important HDF5 tuning decisions.

python

import h5py, numpy as np
with h5py.File('shot.h5', 'w') as f:
    g = f.create_group('diagnostics/interferometer')
    d = g.create_dataset('density', data=np.zeros((10000,)),
                         chunks=(1000,), compression='gzip')
    d.attrs['units'] = 'm^-3'
    d.attrs['sample_rate_hz'] = 1_000_000

Strengths and cautions

HDF5 handles files far larger than memory, supports parallel I/O on clusters, and keeps metadata beside the data. Its cautions are real: the single-file model can be fragile if a write is interrupted, and complex files can lock readers into the HDF5 library. For long-term open archives, many projects pair HDF5 with checksums and clear documentation. See data integrity.

Use in the record

HDF5 is a common container for the per-shot or per-run scientific record because it keeps arrays, units, and provenance attributes together. It sits alongside NetCDF, which is built on the same underlying library, in the scientific data ecosystem.