Conformal Prediction

Conformal prediction wraps any model to produce prediction sets with guaranteed coverage, assuming only that data are exchangeable.

Distribution-free guarantees

Conformal prediction converts a point predictor into an interval or set predictor that contains the true value with a chosen probability, for example 90 percent, over future exchangeable data. The guarantee is finite-sample and distribution-free: it holds regardless of the model or data distribution, requiring only exchangeability.

Split conformal recipe

Kronos motion — pid vs model

Train the model on a training set
Compute nonconformity scores (for example absolute residuals) on a held-out calibration set
Take the (1-alpha) empirical quantile of those scores as the interval half-width
Predict y-hat plus or minus that width for new inputs

python

import numpy as np
resid = np.abs(y_cal - model.predict(X_cal))
n = len(resid)
q = np.quantile(resid, np.ceil((n+1)*(1-alpha))/n, method='higher')
# interval for a new point: [pred - q, pred + q]

Adaptive width

A fixed-width interval ignores that some inputs are harder than others. Normalizing residuals by an estimated local scale, or using conformalized quantile regression, produces intervals that widen where the model is less certain while keeping the marginal coverage guarantee.

What it does and does not promise

Conformal prediction guarantees marginal coverage averaged over inputs, not conditional coverage for every input, and the guarantee breaks if exchangeability fails, as under distribution shift or time-series dependence. Extensions handle covariate shift and sequential data at the cost of extra assumptions.

Why it complements other UQ

Bayesian methods and ensembles produce uncertainty that may be miscalibrated. Conformal prediction can be layered on top to restore a rigorous coverage guarantee, making it a valuable safety net for surrogate predictions used in engineering decisions.