Evaluating a Disruption Predictor (ROC)

Score a binary classifier that flags impending plasma disruptions, sweep the threshold, and build the ROC curve with its area.

Problem

A disruption predictor outputs a risk score; an operator must choose a threshold above which to trigger mitigation. The receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate across all thresholds, and its area (AUC) summarizes ranking quality independent of any single cutoff.

Scores and labels

We have model scores and ground-truth labels (1 = disruption). Sorting by score and sweeping the threshold traces the ROC. AUC equals the probability that a random positive scores higher than a random negative.

import numpy as np
scores=np.array([0.1,0.4,0.35,0.8,0.7,0.2,0.9,0.6])
y=np.array([0,0,1,1,1,0,1,0])
order=np.argsort(-scores); y=y[order]
P=y.sum(); N=len(y)-P
tpr=np.cumsum(y)/P; fpr=np.cumsum(1-y)/N
tpr=np.r_[0,tpr]; fpr=np.r_[0,fpr]
auc=np.trapz(tpr,fpr)
print('AUC',round(auc,3))

Reading it

An AUC of 0.5 means random guessing; 1.0 means perfect separation. For this toy set the AUC comes out around 0.9, indicating the model ranks most disruptions above most safe shots. The operating point is then chosen on the curve to balance missed disruptions (costly) against false alarms (disruptive to operations), a trade-off AUC alone does not decide.

• ROC is threshold-free, so it compares models fairly even under class imbalance in ranking, though precision-recall is better when positives are rare.
• The chosen operating threshold must reflect the real asymmetry between a missed disruption and a false alarm.
• Kronos disruption-prediction studies report AUC alongside warning-time distributions, since a correct-but-late alarm is useless.