Spread: range, variance and standard deviation

Why the centre is not enough

Two classes both have mean marks of 70. In class A the marks are 68, 70, 70, 72, 70. In class B they are 40, 55, 70, 85, 100. Same mean, completely different classes: in A everyone is close to 70, in B some students struggle and some excel. A teacher would plan very differently for each. Spread is the number that tells them apart.

The range

The range is the largest value minus the smallest. Class A has a range of 72 − 68 = 4; class B has 100 − 40 = 60. The range is easy to understand, but it uses only two values, so a single outlier can make it huge.

Deviations

A deviation is how far one value is from the mean: value minus mean. Values above the mean have positive deviations, values below have negative ones. You might think "average the deviations", but they always add up to zero, because the mean is exactly the balance point. So we need another idea.

Variance: the average squared deviation

Squaring each deviation makes it positive and gives big distances extra weight. The variance is the average of those squared deviations. In words: find the mean, find each value's distance from it, square the distances, then average them.

times = [30, 35, 40, 45, 50]   # delivery times in minutes

mean = sum(times) / len(times)                 # 40.0
deviations = [t - mean for t in times]         # [-10.0, -5.0, 0.0, 5.0, 10.0]
squares = [d ** 2 for d in deviations]         # [100.0, 25.0, 0.0, 25.0, 100.0]
variance = sum(squares) / len(squares)
print(variance)            # 50.0
print(variance ** 0.5)     # 7.0710678118654755

Standard deviation

Variance is in squared units (square minutes, square rupees), which nobody can picture. Taking the square root brings it back to the original units. That is the standard deviation (SD): roughly, the typical distance of a value from the mean. Here the SD is about 7.07 minutes, so most deliveries land within about 7 minutes of 40.

Population or sample: n or n − 1

If your data is the whole population (every delivery this month), divide the squared deviations by n, the number of values. If it is a sample used to estimate a bigger population, divide by n − 1. This is called Bessel's correction. The reason: a sample's own mean sits closer to its own points than the true population mean does, so the squared deviations come out a little too small. Dividing by one less makes up for that on average.

import statistics

times = [30, 35, 40, 45, 50]
print(statistics.pvariance(times), statistics.pstdev(times))   # 50 7.0710678118654755
print(statistics.variance(times), statistics.stdev(times))     # 62.5 7.905694150420948

With large data the difference is tiny (dividing by 9,999 or 10,000 hardly matters). With small samples, like 5 values, it is noticeable: 7.07 versus 7.91 above.

Common mistakes

• Comparing averages without spread. Fix: report the mean (or median) together with the SD (or IQR, next topic).
• Forgetting the square root and reporting variance as if it were in the original units. Fix: SD = square root of the variance.
• Mixing population and sample formulas between tools. Fix: check ddof or the function name, and be consistent.
• Trusting the range when there is an outlier. Fix: use the SD or the IQR, which depend on all or the middle values.