Train, validation and test sets

Why hold data back?

A student who memorises last year's question paper may score 100% on it and still fail a new paper. A model can do the same: it can memorise its training examples. The only fair test is on examples it never saw while learning. So before training, we put some data aside and do not touch it until the end.

The three sets

If you tune settings by looking at the test score again and again, the test set slowly becomes part of training, and its score becomes optimistic. That is why the validation set exists. With small datasets, cross-validation (topic 6) replaces a fixed validation set.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Shuffle, but not for time

Random splitting assumes the rows are independent. For time series, like daily sales or stock prices, a random split lets the model "see the future": it trains on next week and is tested on last week. For time data, split by time instead: train on the past (for example, January to September) and test on the later period (October to December). This matches how the model will be used.

Data leakage

Leakage means the training data contains information that would not be available at prediction time, or that comes from the test set. The model then scores brilliantly in testing and fails in real use. It is one of the most common and expensive mistakes in data science.

• Target leakage: a feature is a consequence of the label. Using "refund_issued" to predict "complaint_filed" is cheating.
• Preprocessing leakage: scaling or filling missing values using statistics from ALL the data, including the test set. Fix: fit the scaler on the training set only, then apply it to the test set.
• Duplicate leakage: the same customer or the same photo appears in both training and test sets. Fix: split by customer id (group split).
• Time leakage: using future information, such as next month's average, as a feature.

Common mistakes

• No fixed random_state, so every run gives a different split and nobody can reproduce results.
• Not stratifying a rare class, so the test set ends up with almost no positive cases.
• Peeking at the test set during EDA and feature design. Fix: split early and explore only the training data.
• A test score that is "too good to be true". Treat it as a leakage alarm, not a celebration.