Lossless Data Compression
Lossless compression shrinks data so it can be perfectly reconstructed, exploiting redundancy and predictability in the original.
Smaller, but exactly recoverable
Lossless compression reduces the size of data while guaranteeing that decompression restores the original bit for bit. It works by finding and removing redundancy: repeated patterns, predictable sequences, and skewed symbol frequencies. Because nothing is discarded, it is the only acceptable choice for scientific raw data and the published record.
How it works
- Entropy coding (Huffman, arithmetic): assign shorter codes to frequent symbols.
- Dictionary methods (LZ77, LZW): replace repeated substrings with references.
- Run-length encoding: collapse runs of identical values.
- Delta encoding: store differences between consecutive values, ideal for smooth series.
The entropy limit
Shannon's source coding theorem sets a floor: no lossless method can compress data below its entropy, the average information content per symbol. Data that is already random cannot be compressed at all. Compression ratios therefore reflect how much structure and predictability the data contains, not the cleverness of the algorithm alone.
Common algorithms
- gzip/DEFLATE: general-purpose, widely supported.
- zstd: high ratio with fast decompression, tunable levels.
- LZ4: very fast, modest ratio, good for hot data.
- Specialized codecs for floating-point arrays that exploit numeric structure.
In scientific storage
Array formats such as HDF5 and NetCDF apply lossless compression per chunk, so a smooth simulation field can shrink dramatically while remaining exactly recoverable. Delta encoding is especially effective on slowly varying time series. For the Kronos published record, only lossless compression is used on authoritative data, so a decompressed dataset is provably identical to the deposited original, verifiable by checksum.