Ensembles and Model Averaging
Combining several models to improve accuracy and, importantly, to estimate uncertainty.
Why combine models
An ensemble aggregates the predictions of several models. Averaging reduces variance, so ensembles are usually more accurate and more stable than any single member, especially on the small, noisy datasets common in fusion. The disagreement among members also provides a practical uncertainty estimate.
Ensemble methods
- Bagging: train members on resampled data, then average
- Boosting: train members in sequence, each fixing the last's errors
- Deep ensembles: train several networks from different initializations
- Stacking: learn how to combine members with a meta-model
Uncertainty from disagreement
When ensemble members agree, the prediction is well supported; when they disagree, it is uncertain, often because the input is unlike the training data. This makes deep ensembles a simple and effective source of epistemic uncertainty and a signal for out-of-distribution inputs, both valuable in fusion.
Costs and trade-offs
An ensemble costs several times more to train and run than one model, which matters for real-time control where latency is tight. Techniques like distillation can compress an ensemble's behavior into a single faster model, though some of the uncertainty information is lost. The trade is accuracy and uncertainty against compute.
Honest use
Ensembles reduce variance but do not remove shared bias: if all members learned the same wrong pattern from the data, they will agree confidently and wrongly. Their uncertainty must still be calibrated and checked. Used with that caveat, ensembles are among the most reliable ways to get both better predictions and a usable measure of confidence.