Verification Test Suites
A curated collection of problems with known answers that together exercise every part of a code's numerics.
A Battery of Known Answers
No single test verifies a whole code. A verification test suite is a curated collection of problems, each with a known correct answer, chosen so that together they exercise every numerical component: interior schemes, boundary conditions, source terms, coupling between physics, and special cases. Passing the whole suite is stronger evidence than passing any one test, because different tests catch different bugs.
Sources of Test Problems
- Manufactured solutions, which can be tailored to exercise any term.
- Analytic solutions of simplified problems, which test physically meaningful cases.
- Conservation and symmetry checks, which must hold exactly regardless of the solution.
- Benchmark problems with community reference results.
Coverage
A good suite is designed for coverage, meaning every code path and every term is exercised by at least one test. A term that no test touches is unverified, however well the rest of the code performs. Mapping tests to the code features they exercise reveals gaps, and closing those gaps is an ongoing part of maintaining a verified code.
Maintenance and Regression
A verification suite is only useful if it keeps running. Folded into the regression suite and run under continuous integration, it confirms on every change that the code remains verified. Left to run manually, it decays: tests break, get skipped, and stop reflecting the code. The suite should also grow, because every bug found in the wild should become a new test that would have caught it, so the same defect cannot return unnoticed.
For a design code, the test suite is part of the credibility record. A reviewer can inspect which physics the suite verifies and to what order, and thereby judge how much of the code underlying a design claim actually rests on verified numerics rather than on untested assumption.