Numerical Precision on Hardware
Floating-point formats trade range, accuracy, speed, and memory; choosing the right precision is central to fast and correct computation.
Representing real numbers
Hardware stores real numbers in floating-point formats: a sign, an exponent for range, and a mantissa for precision. The IEEE 754 standard defines the common formats. Because the mantissa has finite width, most real numbers are stored approximately, and every operation can introduce a small rounding error.
The standard formats
- FP64 (double): about 15-16 decimal digits; the default for scientific accuracy
- FP32 (single): about 7 digits; half the memory and often higher throughput
- FP16 / BF16 (half): about 3-4 digits; fast on accelerators, used in ML
- FP8 and integer formats: minimal precision for inference and specialized kernels
Accuracy versus speed
Lower precision uses less memory and less bandwidth and runs faster on tensor units, but carries larger rounding error and narrower range. FP16 in particular overflows and underflows easily; BF16 keeps FP32's exponent range at the cost of mantissa bits, which is why it is favored for training. The choice is a deliberate accuracy-versus-throughput trade.
Error accumulation
Rounding errors can accumulate over long computations. Summing many numbers loses accuracy when magnitudes differ widely; compensated summation (Kahan) recovers much of it. Subtracting nearly equal numbers causes catastrophic cancellation, magnifying relative error. Numerically stable algorithms are designed to limit such effects.
Precision in practice
Scientific solvers often need FP64 for stability, while mixed-precision methods do bulk work in lower precision and correct in higher. Machine learning tolerates low precision because training is inherently noisy. Matching precision to the numerical requirements of each phase is a core performance decision.