Half-Precision Number Formats
FP16 and BF16 are 16-bit floating-point formats that differ in how they split bits between exponent range and mantissa precision.
Two 16-bit formats
Both FP16 and BF16 use 16 bits, but they divide them differently. FP16 (IEEE half) uses 1 sign bit, 5 exponent bits, and 10 mantissa bits. BF16 (bfloat16) uses 1 sign bit, 8 exponent bits, and 7 mantissa bits. FP16 has more mantissa, so it resolves finer differences; BF16 has more exponent, so it spans a far wider dynamic range, matching FP32's range while carrying less precision.
Why range often beats precision
BF16's 8 exponent bits give it the same range as 32-bit float, so values rarely overflow or underflow when converting from FP32. This makes BF16 easy to drop into training pipelines that were built for FP32, since it does not need the loss scaling FP16 often requires to avoid underflowing small gradients. FP16's narrow range (roughly up to 65,504) means large intermediate values can overflow to infinity, which is the price of its extra precision.
- FP16: 5 exponent, 10 mantissa bits, more precision, narrow range.
- BF16: 8 exponent, 7 mantissa bits, wide range, less precision.
- BF16 matches FP32 range, easing conversion and avoiding loss scaling.
- Both are typically multiplied in 16 bits but accumulated in FP32.
Precision loss in numbers
With 10 mantissa bits, FP16 resolves relative differences of roughly one part in a thousand; BF16, with 7 mantissa bits, resolves roughly one part in a hundred. Neither is adequate for accumulating long sums, which is why hardware multiplies in 16 bits but sums the products in FP32. The choice between them depends on whether a workload is limited by range (favor BF16) or by fine precision (favor FP16).
In practice
A neural surrogate trained on Hyperion simulation outputs is usually trained in BF16 for its friendly range, with FP32 accumulation. Any 16-bit inference result feeding a physics decision is checked against an FP32 or FP64 reference before it is trusted.