Overfitting, underfitting and regularisation

The goal is generalisation

We do not care how well a model remembers its training data. We care how well it predicts new cases. That ability is called generalisation. Comparing the training score with the validation or test score is the quickest way to see whether a model generalises.

Bias and variance in plain words

Bias is error from wrong assumptions, like fitting a straight line to a curve: the model is consistently off. Variance is error from being too sensitive to the particular training data: train on a slightly different sample and you get a very different model. Simple models tend to have high bias and low variance; very flexible models the reverse. The bias-variance trade-off is about finding the flexibility that minimises total error on new data.

How to fix underfitting

• Use a more flexible model (for example, trees instead of a line).
• Add better features that carry real signal.
• Reduce regularisation, or train longer.

How to fix overfitting

• Get more training data, the most reliable cure.
• Use a simpler model or limit its complexity (for trees: max_depth, min_samples_leaf).
• Remove noisy or useless features.
• Regularise: add a penalty for large weights to the loss.
• Early stopping: stop training when the validation score stops improving.
• For neural networks: dropout and data augmentation (module 9).

Regularisation: L2 (Ridge) and L1 (Lasso)

Regularisation adds a penalty to the loss so the model prefers small weights. L2 regularisation (Ridge) adds the sum of squared weights; it shrinks all weights smoothly. L1 regularisation (Lasso) adds the sum of absolute weights; it pushes some weights to exactly zero, which also selects features. The strength is set by a hyperparameter, often called alpha (in Ridge and Lasso) or C (in LogisticRegression, where smaller C means stronger regularisation).

from sklearn.linear_model import Ridge, Lasso

ridge = Ridge(alpha=1.0).fit(X_train, y_train)   # shrinks all weights
lasso = Lasso(alpha=0.1).fit(X_train, y_train)   # sets some weights to 0
print((lasso.coef_ == 0).sum(), "features dropped by Lasso")

Learning curves

A learning curve plots training and validation scores as the training set grows. If both are low and close, you are underfitting and more data will not help much; you need a better model or features. If there is a big gap that shrinks as data grows, you are overfitting and more data will help.

Common mistakes

• Celebrating a 100% training score. Fix: always compare with validation.
• Fixing overfitting by tuning endlessly on the test set, which just overfits to the test set.
• Regularising without scaling features first.