Cross-Validation
Hold out part of the data to test a fitted model on cases it did not see, guarding against fitting noise instead of signal.
Testing on Held-Out Data
A model tuned to data will fit that data; the real question is whether it predicts new data. Cross-validation answers this by partitioning the data, fitting on part and testing on the rest, so the model is always evaluated on cases it did not see during fitting. It is the standard guard against overfitting, where a model captures noise rather than the underlying relationship.
K-Fold Cross-Validation
- Split the data into k roughly equal folds.
- For each fold, fit on the other k minus one folds and test on the held-out fold.
- Average the test performance across all folds for an estimate of predictive accuracy.
Leave-One-Out and Its Cost
Taking k equal to the number of data points gives leave-one-out cross-validation, which uses the data most fully but requires refitting the model as many times as there are points. For expensive models this is impractical, and a smaller k, such as five or ten folds, gives a good estimate at a fraction of the cost. The choice trades computational cost against the variance of the estimate.
Where It Fits in V&V
Cross-validation is a tool for the calibration and surrogate-modeling steps of V&V, not a substitute for validation against independent experiments. It tests whether a fitted model generalizes across the available data, which is essential for building trustworthy surrogates and for checking that a calibration is not overfit. But all of that data may share the same regime and the same systematic errors, so passing cross-validation does not establish that the model is right in a new physical regime.
The honest reading: cross-validation confirms a model interpolates its training regime well; genuine validation still requires data from conditions the model was never fit to, ideally from a different experiment. Both are needed, and they answer different questions.