Class Imbalance and Rare Events
Why disruptions and other rare plasma events break naive accuracy, and how to train and evaluate honestly around them.
The imbalance problem
In fusion, the events that matter most are the rarest. Disruptions, specific instabilities, and off-normal diagnostics occupy a tiny fraction of samples. A model that always predicts the majority class can reach high accuracy while being useless.
Better metrics
- Precision and recall, and the precision-recall curve
- True-positive rate at a fixed false-alarm rate
- Area under the ROC curve, read with imbalance in mind
- For prediction: warning-time distribution, not just hit rate
Training techniques
Resampling (oversampling the minority class or undersampling the majority), class-weighted loss functions, and focal loss all shift the model toward the rare class. Synthetic minority oversampling can help but risks generating unphysical samples. Careful thresholding after training often matters more than the training trick used.
Cost asymmetry
A missed disruption can damage hardware; a false alarm wastes a discharge. These costs are not equal, so the operating threshold should reflect the real cost ratio rather than maximizing a symmetric score. This is a decision-theory choice, made with operators, not an ML default.
Evaluation pitfalls
Reporting a single accuracy number, splitting by time slice instead of by shot, or tuning the threshold on the test set all produce misleadingly good numbers. Honest evaluation isolates the rare class, uses forward-in-time splits, and reports uncertainty from the small number of positive examples.
Because a fusion campaign may contain only dozens of disruptions of a given type, confidence intervals on rare-event metrics are wide, and claims should be stated with that in mind.