Gradient Boosting

Gradient boosting builds an additive ensemble of small trees, each fitting the residual errors of those before it.

Learning from mistakes

Gradient boosting builds a model as a sum of many weak learners, usually shallow decision trees, added one at a time. Each new tree is fit to the errors the current ensemble still makes, so the model steadily corrects its own residuals. The result is one of the strongest methods for tabular data.

The gradient view

At each round the algorithm computes the negative gradient of the loss with respect to the current predictions (for squared error these are just the residuals) and fits the next tree to those pseudo-residuals. Adding a scaled version of that tree is one step of functional gradient descent on the loss.

• learning_rate (shrinkage): scales each tree's contribution; smaller values need more trees but generalize better.
• n_estimators: the number of boosting rounds.
• max_depth: keeps each tree weak, typically 3 to 6.
• subsample: stochastic boosting trains each tree on a data fraction for regularization.

from sklearn.ensemble import GradientBoostingClassifier
gb = GradientBoostingClassifier(learning_rate=0.05, n_estimators=400,
                                max_depth=3, subsample=0.8)
gb.fit(X_train, y_train)

Boosting versus bagging

Where random forests reduce variance by averaging independent trees in parallel, boosting reduces bias by adding dependent trees in sequence. Boosting can overfit if run too long, so use early stopping on a validation set and modest learning rates.

Modern implementations such as XGBoost, LightGBM, and CatBoost add regularization, clever tree-building, and speed, and dominate many structured-data competitions.