Calibration of Probabilistic Predictions
A well-calibrated model's stated probabilities match observed frequencies, so a ninety percent interval really contains the truth ninety percent of the time.
The promise behind an error bar
When a model states a ninety percent prediction interval, it makes a testable promise: over many such predictions, the truth should fall inside about ninety percent of the time. Calibration is whether that promise is kept. A model can be accurate on average yet badly calibrated - confidently wrong - which is often more dangerous than plain inaccuracy.
Overconfidence and underconfidence
An overconfident model states intervals that are too narrow, so the truth falls outside more often than claimed. An underconfident model states intervals that are too wide, wasting information. Deep neural networks are notoriously overconfident by default; many surrogates need explicit recalibration before their uncertainty can be trusted.
Diagnostics
- Reliability diagrams plot predicted probability against observed frequency; the diagonal is perfect
- Coverage tests count how often true values fall in stated intervals across a test set
- The probability integral transform should be uniform for a calibrated model
- Sharpness measures how narrow intervals are, subject to staying calibrated
Proper scoring rules
Proper scoring rules grade probabilistic predictions so that honesty is optimal - the best score is earned by reporting your true belief. The continuous ranked probability score, the negative log predictive density, and the Brier score for classification reward predictions that are both sharp and calibrated, penalizing overconfidence and vagueness alike.
Recalibration
When a model is miscalibrated, post-hoc methods fix the reported probabilities without retraining the mean: temperature scaling adjusts the sharpness of classifier outputs, isotonic and Platt scaling remap predicted probabilities, and conformal prediction wraps any model to produce intervals with guaranteed finite-sample coverage under mild assumptions.
Why it is non-negotiable
Uncalibrated uncertainty makes risk-based decisions unreliable. In Kronos UQ, the error bars on surrogate predictions for the machines are checked for calibration on held-out data and recalibrated when needed, so that a stated interval means what it says when it informs a design margin.