Thresholds, ROC-AUC and imbalanced classes

Scores, not just labels

Logistic regression, trees, boosting and neural networks all produce a score or probability first. The yes/no label only appears when you pick a threshold. Changing the threshold changes every count in the confusion matrix, so the "same model" can behave very differently. Evaluating only at 0.5 hides most of what the model can do.

The trade-off

• Lower threshold: more cases flagged → recall goes up, precision usually goes down (more false alarms).
• Higher threshold: fewer cases flagged → precision usually goes up, recall goes down (more misses).

y = [1, 1, 0, 1, 0, 0]
scores = [0.9, 0.7, 0.6, 0.4, 0.3, 0.1]
for th in [0.8, 0.5, 0.35]:
    pred = [1 if s >= th else 0 for s in scores]
    tp = sum(1 for t, p in zip(y, pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y, pred) if t == 0 and p == 1)
    print(th, "precision", round(tp / (tp + fp), 2), "recall", round(tp / 3, 2))

The ROC curve and AUC

The ROC (receiver operating characteristic) curve plots the true positive rate (recall) against the false positive rate (FP ÷ all real negatives) for every possible threshold. A useless model that guesses at random follows the diagonal. A good model bends towards the top-left corner.

AUC is the area under that curve, from 0.5 (random) to 1.0 (perfect). It has a friendly meaning: AUC is the probability that a randomly chosen positive example gets a higher score than a randomly chosen negative one. Because it looks at the ranking of scores, not one threshold, it is good for comparing models before you pick a threshold.

When ROC-AUC flatters: precision-recall curves

When positives are very rare (say 0.1% of payments), the false positive rate stays tiny even with many false alarms, so ROC-AUC can look excellent while precision is poor. The precision-recall (PR) curve and its summary, average precision, focus on the rare class and are often more honest for fraud, disease and defect detection.

Dealing with imbalanced classes

• Pick metrics that look at the rare class: recall, precision, F1, PR-AUC.
• Use class weights, for example class_weight="balanced" in scikit-learn, so mistakes on the rare class cost more during training.
• Resample the training set: oversample the rare class or undersample the common one. SMOTE creates synthetic rare examples by mixing neighbours. Only resample the TRAINING data, never the test set.
• Tune the threshold on validation data to meet a business target, such as "recall at least 90%".

Choosing the threshold

Put a cost on each mistake. If a missed fraud costs ₹20,000 and a false alarm costs ₹50 of call-centre time, try thresholds on validation data and pick the one with the lowest total cost. This turns a technical choice into a business decision everybody can understand.

Common mistakes

• Reporting only ROC-AUC for a very rare class. Fix: add precision-recall numbers.
• Choosing the threshold on the test set. Fix: choose it on validation data, then confirm once on test.
• Balancing the test set by resampling. The test set must look like the real world.