Regression metrics: MAE, MSE, RMSE and R²

Why several metrics?

A delivery-time model that is usually 2 minutes off but sometimes 40 minutes off is very different from one that is always 6 minutes off, yet both may have similar averages. Each regression metric looks at errors in a different way, and choosing one is a statement about which mistakes hurt your business most.

The four metrics

actual = [30, 25, 40, 20]
pred = [28, 27, 30, 21]
errors = [a - p for a, p in zip(actual, pred)]
mae = sum(abs(e) for e in errors) / len(errors)
mse = sum(e * e for e in errors) / len(errors)
print(mae, mse, round(mse ** 0.5, 2))

MAE versus RMSE

If RMSE is much bigger than MAE, a few large errors are present. Pick MAE when all errors cost roughly the same per unit (each minute late is equally annoying). Pick RMSE when big misses are disproportionately costly (being an hour late loses the customer). Report both when unsure.

R² explained

SS_tot is the total squared distance of the actual values from their mean: the error of the simplest baseline. SS_res is the squared error of your model. R² = 1 − SS_res ÷ SS_tot. An R² of 0.8 means the model removes 80% of the baseline's squared error. R² = 0 means no better than the mean, and R² can even be negative on test data if the model is worse than the mean.

Percentage errors

MAPE (mean absolute percentage error) averages |error| ÷ actual. Business teams like it ("we are 8% off on average"), but it breaks when actual values are zero or tiny, and it treats over- and under-prediction unequally. Use it with care, for example for sales of products that always sell some units.

Common mistakes

• Comparing RMSE across different targets or units. Fix: compare models on the same test set and target.
• Reporting R² alone. Fix: add MAE or RMSE in real units.
• Forgetting that MSE is in squared units, so "MSE = 25" means a typical error of about 5, not 25.