Numerical Precision and Floating-Point Data
How numbers are stored, in what precision and format, affects storage size, accuracy, and whether results reproduce exactly.
Numbers have a representation
A real number cannot always be stored exactly in a computer; it is approximated by a finite binary format. The choice of format sets how much precision each value carries, how much space it uses, and how rounding accumulates through a computation. For scientific data, these choices affect both storage and the trustworthiness of results.
Floating-point formats
- Double precision (64-bit): about 15-16 significant decimal digits; the scientific default.
- Single precision (32-bit): about 7 digits; half the storage, faster on some hardware.
- Half precision (16-bit): about 3 digits; used to shrink large arrays or speed machine learning.
- The IEEE 754 standard defines these formats and how arithmetic behaves.
Rounding and error accumulation
Each floating-point operation rounds to the nearest representable value, introducing a tiny error. Over millions of operations these can accumulate, and subtracting nearly equal numbers can amplify them (catastrophic cancellation). Numerical methods are designed to keep such errors bounded, and reporting results honestly means knowing the precision they actually support.
Precision versus storage
Storing data in lower precision than the computation used can shrink archives substantially, but it is a form of lossy reduction and must be a deliberate, documented choice. The authoritative record should preserve the precision the results are valid to; reduced-precision copies are for previews and media, not the citable data.
Precision and reproducibility
Nondeterminism in parallel floating-point arithmetic, where the order of summation varies between runs, can make bit-for-bit reproduction impossible even with identical inputs. This is why a rigorous record states whether reproduction is byte-exact or tolerance-based, and gives the tolerance. For a fusion program's deposited results, frozen physics values such as the breeder Q of 3.424 are reported at a precision the simulations support, and reproduction is checked within a stated bound. See reproducible datasets.