Boosting
Boosting turns weak learners into a strong one by training them in sequence, each focusing on the previous errors.
Sequential correction
Boosting builds a strong model from many weak ones, each only slightly better than chance, by adding them in sequence. Every new learner concentrates on the examples its predecessors got wrong. The final prediction is a weighted combination of all of them. Unlike bagging's parallel averaging, boosting is inherently sequential.
AdaBoost, the original idea
AdaBoost keeps a weight on each training example. After each weak learner, it raises the weight of misclassified points and lowers the weight of correct ones, so the next learner focuses on the hard cases. Each learner also earns a say in the vote proportional to its accuracy.
Gradient boosting, the modern form
- Frames boosting as gradient descent in function space.
- Each learner fits the negative gradient of the loss (the residual for squared error).
- A learning rate shrinks each contribution to improve generalization.
- Powers XGBoost, LightGBM, and CatBoost.
See gradient boosting and XGBoost for the details.
Trade-offs
Boosting reduces bias and often reaches the best accuracy on tabular data, but because each learner chases residuals it can overfit noisy labels and outliers if run too long. Guard against this with a small learning rate, shallow base learners, and early stopping on a validation set. Boosting is harder to parallelize than bagging because of its sequential dependence.