Mixed-Precision Computing
Mixed precision does most work in fast low precision and selectively refines in higher precision, capturing speed without sacrificing accuracy.
Best of both
Mixed-precision computing combines formats within one algorithm: the bulk of the arithmetic runs in a fast, low-precision format while critical accumulations or corrections use higher precision. The aim is the throughput and reduced memory traffic of low precision with accuracy close to a full high-precision computation.
Iterative refinement
A classic technique solves a linear system in low precision, computes the residual in high precision, and iterates a small correction. The expensive factorization runs fast in low precision; the residual, done in high precision, restores accuracy. Only a few refinement steps recover a full-precision answer for many well-conditioned problems.
In machine learning
Deep-learning training routinely stores weights and does matrix multiplies in FP16 or BF16 on tensor units, while keeping a high-precision master copy of the weights and accumulating in FP32. Loss scaling prevents small gradients from underflowing in FP16. This gives large speedups with negligible effect on final model quality.
Where it pays
- Memory-bound kernels: halving bytes moved roughly doubles effective bandwidth
- Tensor-unit workloads: low-precision matrix multiply is far faster
- Large models: smaller weights fit more in limited accelerator memory
When to be careful
Mixed precision is not free correctness. Ill-conditioned problems, wide dynamic ranges, and long serial dependency chains can amplify low-precision error beyond what refinement recovers. The discipline is to keep the numerically sensitive operations in high precision and verify results against a full-precision reference before trusting the speedup.