Calibration of Probabilistic Models
Making predicted probabilities and uncertainties mean what they say, so decisions can rely on them.
What calibration means
A model is calibrated when its stated confidence matches its actual accuracy. Among predictions it makes with 80 percent confidence, about 80 percent should be correct. A disruption predictor that outputs a probability is only useful for setting alarm thresholds if that probability is calibrated.
Why models miss it
Modern neural networks are often overconfident: they output high probabilities even when wrong, especially on data unlike their training set. Class imbalance, distribution shift, and overfitting all worsen calibration. A miscalibrated model can look accurate on average while giving dangerously confident wrong answers.
Measuring calibration
- Reliability diagrams: plot predicted versus observed frequency
- Expected calibration error: average gap across confidence bins
- Proper scoring rules such as the Brier score and log loss
Fixing it
Post-hoc methods recalibrate a trained model on held-out data: temperature scaling adjusts the sharpness of probabilities, and isotonic or Platt scaling remap them. These are simple and effective for classification. For regression uncertainty, quantile calibration and conformal methods play the analogous role.
Why it matters for decisions
Alarm thresholds, control caution, and design margins all consume probabilities. If those probabilities are wrong, the decisions built on them are wrong in ways that look justified. Calibration is checked before deployment and monitored afterward, because it can drift as operating conditions change. A calibrated model is a precondition for trusting its numbers, not an optional refinement.