Cross-Validation for Surrogates
Cross-validation reuses scarce data by rotating holdout folds, estimating a surrogate's out-of-sample error without new expensive runs.
Making the most of few samples
Surrogates for expensive models are trained on small datasets, so setting aside a large test set is wasteful. Cross-validation solves this by systematically rotating which data serves as the test set, so every point is used for both training and testing across different splits. The averaged error estimates out-of-sample accuracy without spending new evaluations.
k-fold cross-validation
The data is split into k equal folds. The surrogate is trained on k-1 folds and tested on the held-out fold, repeated so each fold is held out once. The k test errors are averaged. Common choices are five or ten folds, balancing bias (few folds train on too little data) against variance and cost (many folds are expensive).
Leave-one-out
Leave-one-out cross-validation takes k equal to the number of samples: each point is predicted from all the others. It uses the most training data per fit and is nearly unbiased, but naively it requires N refits. For linear-in-parameters models and Gaussian processes, an algebraic shortcut computes all leave-one-out errors from a single fit, making it cheap.
What it is used for
- Estimating a surrogate's predictive error before deployment
- Selecting hyperparameters such as kernel length scales or RBF shape
- Comparing candidate surrogate types on the same data
- Driving adaptive sampling toward high-error regions
Pitfalls
Cross-validation assumes samples are independent; correlated or clustered data - such as points along a design trajectory - inflates optimism unless folds respect the structure. It also tends to be pessimistic for the final model, which trains on all data. And repeatedly tuning to cross-validation error can leak information, so a final untouched test set is still valuable when affordable.
In practice
Cross-validation is the routine health check on Kronos surrogates: leave-one-out error guides kernel and hyperparameter choices for models of the machines and flags where the surrogate is unreliable, so adaptive sampling can reinforce those regions.