Correlation: how two things move together

Why look at two columns together

So far each summary described one column. Many real questions are about pairs: do students who study more score higher? Do houses closer to a metro station cost more? Does more advertising bring more sales? The first step is a scatter plot, and the first number is the correlation.

Scatter plots

A scatter plot puts one column on the x-axis and the other on the y-axis, with one dot per row. If the dots rise from left to right, the link is positive; if they fall, negative; if they form a shapeless cloud, there is no clear link. Always look at the plot before trusting any number.

import matplotlib.pyplot as plt

hours = [1, 2, 3, 4, 5]
marks = [52, 55, 61, 64, 70]
plt.scatter(hours, marks)
plt.xlabel("Hours studied per day")
plt.ylabel("Marks")
plt.show()

Covariance

Covariance measures whether two columns move in the same direction. For each row, take x's deviation from its mean times y's deviation from its mean. When both are above their means (or both below), the product is positive; when one is above and the other below, it is negative. Average these products (dividing by n − 1 for a sample) and you get the covariance. Positive means "move together", negative means "move opposite".

The problem: covariance has strange units (hours × marks) and its size changes if you switch from rupees to thousands of rupees. So you cannot tell from it whether a link is strong or weak.

Pearson's correlation coefficient r

Pearson's r fixes that by dividing the covariance by the two standard deviations: r = covariance / (sd of x × sd of y). The units cancel, and r always lands between −1 and +1. Use the same divisor (n or n − 1) for the covariance and both SDs, and it cancels out, so r is the same either way.

hours = [1, 2, 3, 4, 5]
marks = [52, 55, 61, 64, 70]
n = len(hours)
mx = sum(hours) / n   # 3.0
my = sum(marks) / n   # 60.4

cov = sum((x - mx) * (y - my) for x, y in zip(hours, marks)) / (n - 1)
sx = (sum((x - mx) ** 2 for x in hours) / (n - 1)) ** 0.5
sy = (sum((y - my) ** 2 for y in marks) / (n - 1)) ** 0.5
print(round(cov, 2))             # 11.25
print(round(cov / (sx * sy), 3)) # 0.993

import statistics

print(round(statistics.correlation(hours, marks), 3))   # 0.993
print(statistics.covariance(hours, marks))              # 11.25

r measures straight lines only

r only detects straight-line (linear) patterns. A perfect U-shaped curve can have r = 0. And a single outlier can push r up or down a lot. That is why you always draw the scatter plot first.

xs = [-2, -1, 0, 1, 2]
ys = [x * x for x in xs]   # [4, 1, 0, 1, 4]: a perfect U shape
print(statistics.correlation(xs, ys))   # 0.0

Correlation is not causation

Ice-cream sales and drowning deaths rise and fall together. Ice-cream does not cause drowning. Hot weather drives both: more people buy ice-cream and more people swim. A hidden factor that drives both variables is called a confounder. In India, umbrella sales and dengue cases both rise in the monsoon, and the rain is the confounder.

• Confounder: a third factor causes both (summer heat, the monsoon).
• Reverse cause: the arrow points the other way (hospitals have many sick people, but hospitals do not make people sick).
• Chance: with enough columns, some pairs correlate by pure luck.

Common mistakes

• Saying "X causes Y" from a correlation. Fix: say "X and Y are associated", and list possible confounders.
• Trusting r without a scatter plot. Fix: always plot; look for curves, clusters and outliers.
• Reading r = 0 as "no relationship". Fix: it means no straight-line relationship; there may be a curve.
• Computing r on tiny samples. Fix: with 5 points, r can be large by chance; collect more data.