Regression Testing
Automated tests that catch when a change silently breaks something that used to work, the immune system of a long-lived code.
Guarding Against Backsliding
A regression is a defect introduced into functionality that previously worked. In a scientific code that evolves for years, an innocuous change to one routine can perturb results far away. Regression testing runs a fixed suite of tests after every change and flags any output that has drifted from its recorded correct value, so the break is caught immediately rather than discovered months later in a paper.
What a Regression Test Records
- A defined input case and the correct output, captured when the code was known good.
- A tolerance, because floating-point results rarely reproduce to the last bit across platforms.
- A clear pass or fail verdict, so the test can run unattended.
Choosing Tolerances
Setting the comparison tolerance is the subtle part. Too tight and the test fails on harmless round-off differences between compilers or hardware; too loose and it misses real regressions. A good practice ties the tolerance to the known numerical accuracy of the case: the test should pass for changes within the discretization error and fail for anything larger.
Levels of Test
Regression suites are layered. Fast unit tests on individual functions run on every commit. Slower integration tests on full physics cases run nightly. Expensive full-system reproductions run on a schedule or before releases. Layering keeps the fast feedback loop fast while still exercising the expensive paths regularly.
Regression testing is what makes a verified code stay verified. A single order-of-accuracy test proves the code is correct today; a regression suite that includes that test proves it is still correct after every subsequent change. Combined with continuous integration, it converts verification from a one-time event into a standing property of the code base.