Round-off Error
Finite-precision arithmetic cannot represent most real numbers exactly; the accumulated rounding sets a floor on achievable accuracy.
The Precision Floor
Computers store real numbers in finite precision, typically 64-bit double precision with about sixteen significant decimal digits. Almost every real number is rounded to the nearest representable value, and each arithmetic operation can introduce a tiny error. Individually negligible, these errors accumulate and set a floor below which no amount of mesh refinement can push the total error.
Machine Epsilon
The relative gap between one and the next representable number is machine epsilon, roughly two times ten to the minus sixteen for double precision. It bounds the relative error of a single rounded operation. The practical accuracy floor of a calculation is usually some multiple of machine epsilon that grows with the number and type of operations.
Where It Bites
- Subtracting two nearly equal numbers, which cancels significant digits and amplifies relative error.
- Summing many terms of widely different magnitude, where small terms are lost.
- Very fine meshes, where discretization error has fallen to the round-off floor and refinement stops helping.
Seeing It in Verification
Round-off error announces itself at the fine end of a convergence study. As the mesh refines, discretization error falls until it meets the round-off floor; further refinement then makes the error stagnate or even rise, and the observed order of accuracy collapses. Mistaking this floor for a code bug is a common error, as is refining past it and wasting computation. Recognizing the signature keeps the convergence study interpretable.
Mitigations include compensated summation algorithms, reformulating expressions to avoid cancellation, and, where justified, extended precision. In the error budget, round-off is usually a small line item, but it must be checked rather than assumed negligible, because in ill-conditioned problems it can dominate.