Random forests and bagging

The wisdom of crowds

Ask one person to guess the weight of a pumpkin at a village fair and they may be far off. Ask 500 people and average their guesses, and the result is often surprisingly close, because individual errors point in different directions and cancel. A random forest applies this to trees: many slightly different, imperfect trees vote together.

Bagging: bootstrap aggregating

A bootstrap sample is drawn from the training data with replacement: pick a random row, put it back, pick again, until you have as many rows as the original. Some rows appear twice or more, and about 37% do not appear at all. Train one tree on each bootstrap sample, then aggregate: majority vote for classification, average for regression. This is bagging, and it mainly reduces variance (overfitting).

import random

rng = random.Random(0)
rows = list(range(10))
sample = [rng.choice(rows) for _ in rows]
print(sorted(sample))
print(len(set(sample)), "distinct rows out of 10")

The extra randomness: feature subsets

If one feature is very strong, every bagged tree would split on it first and the trees would look alike, so their errors would not cancel. A random forest also picks a random subset of features at each split (often √(number of features) for classification). This makes the trees more different from each other, which makes the vote stronger.

Out-of-bag (OOB) error

Each tree never saw its out-of-bag rows, so they can be used to test it. Combining these gives the OOB score, a free estimate of test performance without a separate validation set. In scikit-learn, pass oob_score=True.

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators=300, max_features="sqrt", oob_score=True, n_jobs=-1, random_state=0)
forest.fit(X_train, y_train)
print(round(forest.oob_score_, 3))
print(sorted(zip(forest.feature_importances_, X_train.columns), reverse=True)[:5])

Feature importance, with a warning

Forests report how much each feature reduced impurity across all trees. This is handy for a first look, but it favours features with many possible split points (like ids or continuous numbers) and can split credit between correlated features. Permutation importance (topic 5) is usually more trustworthy.

Strengths and weaknesses

• Strengths: accurate out of the box, hard to badly overfit by adding trees, little tuning, handles mixed feature types.
• Weaknesses: slower and larger than one tree, harder to explain than a single tree, cannot extrapolate (it averages trees).

Common mistakes

• Thinking more trees cause overfitting. More trees mostly just cost time; the score levels off.
• Treating impurity-based importance as causal. It shows what the model used, not what causes the outcome.
• Including an id column. The forest may find accidental patterns in it.