Decision trees

Why trees?

People already decide with trees: "If the applicant has a regular income, check their existing loans; if those are low, approve." A decision tree learns such a flowchart from data. It handles numbers and categories, needs no scaling and can be drawn on paper, which is why banks, insurers and doctors like it.

The words

• Node: one question, such as "age < 30?".
• Root: the first question at the top.
• Branch: the path for one answer (yes or no).
• Leaf: an end point that gives the prediction (the majority class, or the average value for regression).
• Depth: how many questions lie between the root and the deepest leaf.

Choosing a split: purity

At each node, the tree tries many possible questions and picks the one that splits the data into groups that are as "pure" as possible, meaning each group is mostly one class. Gini impurity measures how mixed a group is: Gini = 1 − Σ pᵢ², where pᵢ is the share of class i. A pure group has Gini 0. A 50/50 mix of two classes has Gini 0.5.

def gini(labels):
    n = len(labels)
    return 1 - sum((labels.count(c) / n) ** 2 for c in set(labels))

print(gini(["yes", "yes", "no", "no"]), gini(["yes", "yes", "yes"]))

For a split, compute the Gini of each side and take the weighted average by group size. The best split has the lowest weighted Gini; the drop from the parent's Gini is the information gain. Entropy (−Σ p·log₂ p) is an alternative measure that usually picks very similar splits.

Splitting on a number

For a numeric feature, the candidate thresholds are usually the midpoints between neighbouring sorted values. With incomes 20, 30, 50 and 70 thousand, the tree tries 25, 40 and 60, and keeps the threshold with the lowest weighted Gini.

Growing and stopping

The tree repeats the split search on each new group, again and again. Left alone, it keeps splitting until every leaf is pure, which usually means memorising noise. Limits stop this: max_depth caps the number of questions, min_samples_leaf requires every leaf to have enough examples, and pruning removes weak branches afterwards.

from sklearn.tree import DecisionTreeClassifier, export_text

tree = DecisionTreeClassifier(max_depth=3, min_samples_leaf=20, random_state=0)
tree.fit(X_train, y_train)
print(export_text(tree, feature_names=list(X_train.columns)))
# |--- income <= 42000.00
# |   |--- existing_loans <= 1.50
# |   |   |--- class: approve
# ...

Strengths and weaknesses

• Strengths: readable, no scaling needed, handles non-linear patterns and interactions ("young AND low income").
• Weaknesses: unstable (a small change in data can change the whole tree), overfits easily, and predictions are step-shaped, so it cannot extrapolate beyond the training range.

Common mistakes

• Leaving max_depth unlimited. Fix: tune depth and leaf size with cross-validation.
• Reading one tree's structure as "the truth" about the data. A slightly different sample may give a different tree.
• Expecting a tree to predict a house price above any price it saw in training. It cannot.