One Gradient-Boosting Round

Fit a shallow tree to the residuals of a constant predictor and watch a single boosting round shrink the loss.

Problem

Gradient boosting builds an additive model by repeatedly fitting a weak learner to the negative gradient of the loss. For squared error the negative gradient is simply the residual, so each round fits a small tree to what the current model gets wrong.

First model and residuals

Start with the mean of the targets as the base prediction. The residuals are target minus mean. A depth-1 tree (a decision stump) splits the feature to reduce residual variance, and we add a shrunken version of its predictions.

import numpy as np
x=np.array([1,2,3,4,5,6]); y=np.array([1,1,2,6,7,7.])
F=np.full_like(y,y.mean())     # base learner = mean = 4.0
r=y-F                          # residuals
# stump: best split threshold minimizing residual SSE
best=None
for thr in [2.5,3.5,4.5]:
    l=r[x<=thr].mean() if (x<=thr).any() else 0
    rr=r[x>thr].mean() if (x>thr).any() else 0
    pred=np.where(x<=thr,l,rr)
    sse=((r-pred)**2).sum()
    if best is None or sse<best[0]: best=(sse,thr,l,rr)
sse,thr,l,rr=best; lr=0.5
F=F+lr*np.where(x<=thr,l,rr)
print('split at',thr,'new SSE',round(((y-F)**2).sum(),2))

Result

The stump splits between the low group (x<=3) and the high group (x>3), predicting negative residuals below and positive above. Adding half of that correction cuts the sum of squared errors sharply. Repeating the round on the new residuals continues to reduce error, each tree correcting the last.

• The learning rate (shrinkage) trades rounds for stability; small rates with many trees usually generalize best.
• Because each learner targets residuals, boosting can overfit if run too long without early stopping.
• Kronos uses gradient-boosted trees on tabular diagnostic features where they often beat deep nets on small datasets.