Confusion matrix, accuracy, precision, recall and F1

Why accuracy is not enough

Imagine 1,000 payments, of which 10 are fraud. A lazy model that says "not fraud" every time is 99% accurate and catches zero fraud. Accuracy hides this because the rare class barely affects it. To see what a classifier really does, we count each kind of right and wrong answer.

The confusion matrix

"Positive" means the class we are looking for: fraud, disease, spam. It is not "good". A false positive is a false alarm; a false negative is a miss.

The metrics

• Accuracy = (TP + TN) ÷ all. The share of all predictions that are right.
• Precision = TP ÷ (TP + FP). Of everything we flagged, how much was real? High precision means few false alarms.
• Recall (also called sensitivity or true positive rate) = TP ÷ (TP + FN). Of all real positives, how many did we catch? High recall means few misses.
• F1 = 2 × precision × recall ÷ (precision + recall). The harmonic mean: it is high only when BOTH are high.

tp, fp, fn, tn = 8, 12, 2, 978
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print(round(precision, 2), round(recall, 2), round(f1, 2))

Which one matters?

It depends on the cost of each mistake. In cancer screening, missing a case (FN) is far worse than an extra check (FP), so recall comes first. In a spam filter, hiding an important email (FP) is worse than letting one spam through, so precision comes first. In fraud detection, both matter, and teams often fix a minimum recall and then maximise precision. Talk to the people who live with the mistakes before choosing.

from sklearn.metrics import confusion_matrix, classification_report

print(confusion_matrix(y_test, y_pred))        # [[TN FP]
                                               #  [FN TP]]
print(classification_report(y_test, y_pred))   # precision, recall, F1 per class

Common mistakes

• Reporting accuracy alone on imbalanced data. Fix: always show the confusion matrix and recall/precision for the rare class.
• Dividing by zero when the model never predicts positive (precision is undefined). Fix: decide a rule; libraries usually report 0.
• Mixing up scikit-learn's matrix layout with a textbook layout. Fix: print the labels.
• Calling the averaged F1 of all classes "the F1" without saying which average (macro or weighted).