Cross-validation and tuning hyperparameters

Why one validation split is not enough

With a small dataset, the score on one validation split depends a lot on which rows happened to land there. Run again with a different split and the score may jump by several points. You might pick a "winning" model that only won by luck. Cross-validation averages over several splits to reduce this luck.

k-fold cross-validation

1. Split the training data into k equal parts called folds (k = 5 or 10 is common).
2. For each fold: train on the other k − 1 folds and score on this fold.
3. Average the k scores, and look at their spread too.

Every example is used for validation exactly once and for training k − 1 times. If 10 rows are split into 3 folds, the folds get 4, 3 and 3 rows: when the rows do not divide evenly, scikit-learn gives the first n % k folds one extra row.

from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

scores = cross_val_score(DecisionTreeClassifier(max_depth=4), X_train, y_train, cv=5, scoring="f1")
print(scores.round(3), scores.mean().round(3))   # e.g. [0.71 0.68 0.74 0.70 0.69] 0.704

Stratified and grouped folds

For classification, StratifiedKFold keeps the class balance in each fold (scikit-learn uses it by default for classifiers when you pass cv=5). If the same customer or patient appears in several rows, use GroupKFold so all their rows stay in one fold; otherwise the model is tested on people it has already seen. For time series, use TimeSeriesSplit, which always trains on earlier data and validates on later data.

Parameters versus hyperparameters

Parameters are learned during training: the weights of a regression, the split points of a tree. Hyperparameters are chosen by you before training: the tree's max_depth, the number of neighbours k in k-NN, the regularisation strength alpha, the learning rate. Tuning means trying hyperparameter values and keeping the best by cross-validated score.

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=0),
    param_grid={"max_depth": [3, 5, 8], "min_samples_leaf": [1, 5, 20]},
    cv=5, scoring="f1",
)
grid.fit(X_train, y_train)
print(grid.best_params_, round(grid.best_score_, 3))
print(grid.score(X_test, y_test))   # final check, once

Keep the test set out of tuning

Cross-validation runs on the training data. The test set is touched once, after you have chosen the model and its hyperparameters. If you tune with cross-validation and then also choose between final candidates by their test score, you have tuned on the test set again. For research-grade estimates, nested cross-validation puts the whole tuning loop inside an outer CV loop.

Common mistakes

• Preprocessing (scaling, filling) on all data before CV, which leaks across folds. Fix: put preprocessing in a Pipeline so it is refitted inside each fold (module 8).
• Using plain k-fold for time series. Fix: TimeSeriesSplit.
• Reporting only the mean CV score. Fix: also report the spread; a mean of 0.80 with scores from 0.60 to 0.95 is unstable.
• Huge grids that take days. Fix: start coarse, then zoom in, or use randomised search.