Gradient boosting

From voting to correcting

A random forest builds independent trees in parallel and averages them. Boosting works in sequence: start with a simple prediction, see where it is wrong, and train the next tree to correct exactly those errors. Then repeat hundreds of times. Each tree is usually small (depth 3 to 8), a "weak learner", but together they become very strong.

The steps for regression

1. Start with a constant prediction for everyone, usually the mean of y.
2. Compute residuals: actual minus current prediction. These are what is still wrong.
3. Fit a small tree to predict the residuals from the features.
4. Update: prediction = prediction + learning_rate × (tree's prediction of the residual).
5. Repeat steps 2–4 for many rounds.

ys = [10, 20, 30]
pred = [sum(ys) / len(ys)] * len(ys)       # start at the mean: 20
lr = 0.5
for round_no in range(2):
    residuals = [y - p for y, p in zip(ys, pred)]
    # pretend the tree predicts each residual perfectly
    pred = [p + lr * r for p, r in zip(pred, residuals)]
    print(round_no + 1, pred)

The name "gradient" comes from the fact that the residual is the negative gradient of the squared-error loss. For other losses (like log loss for classification), the trees fit the gradient of that loss instead. That is why the same method works for many problems.

The key settings

• n_estimators: the number of trees (rounds).
• learning_rate: how much each tree counts, often 0.01 to 0.3. Smaller rates need more trees but usually generalise better.
• max_depth (or num_leaves in LightGBM): how complex each tree is.
• subsample and colsample: using a random share of rows and features per tree, which adds forest-style randomness.
• Early stopping: stop adding trees when the validation score stops improving.

The popular libraries

XGBoost, LightGBM and CatBoost are widely used open-source gradient boosting libraries; scikit-learn also has HistGradientBoostingClassifier and HistGradientBoostingRegressor. They are fast, handle missing values, and are common winners on tabular problems such as credit scoring, churn and demand forecasting. CatBoost is known for handling categorical features well.

from sklearn.ensemble import HistGradientBoostingClassifier

gb = HistGradientBoostingClassifier(learning_rate=0.05, max_iter=500, early_stopping=True, random_state=0)
gb.fit(X_train, y_train)
print(gb.n_iter_, round(gb.score(X_test, y_test), 3))

Common mistakes

• A high learning rate with many trees, which overfits quickly. Fix: lower the rate and use early stopping.
• Tuning everything at once. Fix: start with defaults, then tune learning rate and number of trees, then tree size.
• Skipping a simple baseline. Boosting is powerful, but always check how much it beats logistic regression or a forest.