Temperature Scaling

Temperature scaling recalibrates a neural network confidence with a single learned parameter, without changing predictions.

One knob for confidence

Deep classifiers are usually overconfident: their softmax probabilities are too extreme relative to how often they are correct. Temperature scaling is a minimal fix. It divides the pre-softmax logits by a single positive scalar T before applying the softmax, softening (T greater than one) or sharpening (T less than one) the output distribution.

Why it preserves accuracy

Dividing all logits by the same constant does not change which logit is largest, so the predicted class never changes. Temperature scaling therefore adjusts calibration without touching accuracy. It has exactly one parameter, which makes it robust and data-efficient compared to multi-parameter alternatives.

Fitting the temperature

T is chosen on a held-out validation set by minimizing negative log-likelihood (or equivalently optimizing a calibration objective) with the network frozen. Because there is only one parameter, this is a quick one-dimensional optimization. A learned T greater than one, common for large networks, spreads probability mass away from the top class and pulls confidence back toward reality.

# Optimize T on validation logits
import torch
T = torch.ones(1, requires_grad=True)
opt = torch.optim.LBFGS([T], lr=0.01, max_iter=50)
def step():
    opt.zero_grad()
    loss = nll(logits / T, labels)
    loss.backward(); return loss
opt.step(step)

Scope and limits

Temperature scaling assumes the miscalibration is a uniform over-sharpening across classes, which holds well for many image and text classifiers. It cannot fix class-dependent miscalibration, where richer methods like isotonic or vector scaling help. And like all post-hoc calibration, it is fit on in-distribution data, so it can degrade under distribution shift and should be revalidated when the input distribution moves.