Ensemble Methods
Ensembles combine many models so their errors cancel, usually beating any single model through bagging, boosting, or stacking.
Wisdom of many models
An ensemble combines the predictions of several models into one. When individual models make different, partly independent errors, aggregating them cancels noise and yields a more accurate, more stable predictor than any member. Nearly every top result on tabular data uses an ensemble of some kind.
Three main strategies
- Bagging: train many models on bootstrap samples in parallel and average them; reduces variance (random forests).
- Boosting: train models in sequence, each fixing the last one's errors; reduces bias (gradient boosting, XGBoost).
- Stacking: train diverse base models, then a meta-model learns to combine their outputs.
- Voting: simple majority (hard) or averaged-probability (soft) combination of independent models.
Why diversity is the point
Averaging helps only when members are decorrelated. If all models make the same mistakes, combining them changes nothing. Diversity comes from different training samples (bagging), different features, different algorithms, or different random seeds. The error of an averaged ensemble drops with the members' independence, not merely their count.
Stacking in practice
Stacking trains base learners, generates their out-of-fold predictions to avoid leakage, and feeds those as features to a simple meta-learner such as logistic regression. It can squeeze extra accuracy from complementary models but adds complexity and risk of overfitting the meta-layer.
The costs of ensembles are more computation, more memory, and less interpretability. When a single model is accurate enough and must be explainable, prefer it; when raw accuracy dominates, ensembles usually win. See boosting and random forests for the two most common forms.