Cross-Validation for Fusion Data
Estimating how a model will generalize when data are scarce, correlated, and grouped by shot.
Why validation is subtle here
Cross-validation estimates out-of-sample performance by repeatedly training on part of the data and testing on the rest. In fusion, naive application leaks information and inflates scores, because samples within a shot are highly correlated and campaigns drift over time.
Group-aware splitting
Time slices from one discharge are not independent. If some slices of a shot are in training and others in test, the model can memorize the shot and appear accurate. Splitting by shot (group k-fold) keeps all of a shot on one side, giving an honest estimate of performance on unseen shots.
- Group by shot to prevent within-shot leakage
- Split forward in time for anything used in real time
- Leave-one-machine-out to estimate cross-machine transfer
Temporal validity
For a real-time predictor, the model may only use past data. Random splits violate this by letting future information into training. Time-ordered splits, training on earlier shots and testing on later ones, respect causality and reveal drift over a campaign.
Small-data caveats
With few shots, and few positive examples of rare events, cross-validation estimates are themselves uncertain. Report confidence intervals across folds, not a single number. A difference between two models smaller than the fold-to-fold variation is not evidence one is better.
The payoff
Rigorous cross-validation is what separates a model that will hold up in practice from one that only looked good. In a field where over-claiming is easy and hardware is scarce, this discipline is the foundation of credible results.