Deep Ensembles

Training several neural networks from different random initializations and combining their predictions gives a simple, strong estimate of model uncertainty.

The recipe

A deep ensemble trains M neural networks independently, each with a different random initialization and data shuffling, then averages their predictions. The spread across members estimates epistemic uncertainty: where the networks disagree, the model is unsure. Lakshminarayanan and colleagues showed this simple approach often matches or beats more elaborate Bayesian methods for calibration.

Predictive distribution

Kronos motion — training from sim

For regression, each network outputs a mean and variance; the ensemble mixture has mean equal to the average of member means and total variance equal to the average member variance (aleatoric) plus the variance of the member means (epistemic). This clean decomposition is a practical advantage.

python

import numpy as np
means = np.stack([m.predict_mean(X) for m in models])   # (M, N)
vars_ = np.stack([m.predict_var(X) for m in models])    # (M, N)
mu = means.mean(0)
aleatoric = vars_.mean(0)
epistemic = means.var(0)
total_var = aleatoric + epistemic

Why it works

Different initializations converge to different modes of the loss landscape, so members make different errors far from the training data. Their disagreement grows in extrapolation regions, producing the widening uncertainty a single network cannot express. The averaging also improves point accuracy.

Costs and choices

Training and inference cost scale with the number of members (typically 5 to 10)
Members should differ: random init is essential, bootstrap resampling optional
Proper scoring rules such as negative log-likelihood should guide member training

Limitations

Deep ensembles are not fully Bayesian and can remain overconfident far outside the training distribution, since all members may fail the same way if the architecture imposes a shared inductive bias. They should be validated with calibration diagnostics and, for safety-critical use, combined with conformal prediction to obtain finite-sample coverage guarantees.