Percentiles, quartiles and outliers

Why positions matter

The mean and SD summarise the whole column, but people often ask "where do I stand?". If you scored 72 in an exam, that alone says little. If 90% of students scored at or below 72, you did very well. Positions in sorted data, called percentiles, answer that kind of question, and they are not thrown off by outliers.

Percentiles and percentile rank

The p-th percentile is a value that about p percent of the data is at or below. The 50th percentile is the median. The other direction is the percentile rank of a value: what percent of the data is at or below it. JEE Main, for example, reports an NTA percentile, which is roughly the percentage of candidates in your session who scored equal to or less than you. So a percentile is about your position, not your marks.

marks = [35, 42, 50, 55, 61, 64, 70, 72, 81, 95]
x = 72
at_or_below = sum(1 for m in marks if m <= x)
print(100 * at_or_below / len(marks))   # 80.0

Quartiles and the IQR

Quartiles cut sorted data into four equal parts. Q1, the first quartile, is the 25th percentile; Q2 is the median; Q3 is the 75th percentile. The interquartile range, IQR = Q3 − Q1, is the spread of the middle 50% of the data. Because it ignores the top and bottom quarters, one extreme value cannot blow it up, unlike the range and the SD.

import statistics

times = [22, 25, 27, 28, 30, 31, 33, 35, 36, 40, 95]   # delivery minutes, sorted
q1, q2, q3 = statistics.quantiles(times, n=4)
print(q1, q2, q3)   # 27.0 31.0 36.0
print(q3 - q1)      # 9.0

The 1.5 × IQR rule for outliers

A common rule, from the statistician John Tukey, flags a value as a possible outlier if it is below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR. These two limits are called the fences. Here the fences are 27 − 13.5 = 13.5 and 36 + 13.5 = 49.5, so the 95-minute delivery is flagged.

iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print(low, high)                                  # 13.5 49.5
print([t for t in times if t < low or t > high])  # [95]

Box plots

A box plot (box-and-whisker plot) draws these numbers. The box runs from Q1 to Q3 with a line at the median. The whiskers reach out to the furthest values that are still inside the fences, and points beyond the fences are drawn one by one as outliers. Box plots make it easy to compare spread across groups, such as delivery times per city.

  22         27     31     36       40                     95
   |----------[======|======]--------|                       o
 whisker      Q1   median   Q3    whisker                outlier

import matplotlib.pyplot as plt

plt.boxplot(times)   # whiskers use 1.5 × IQR by default
plt.ylabel("Delivery time (minutes)")
plt.show()

Careful: tools disagree on quartiles

When a quartile falls between two data values, tools use different rules to pick a number in between. None is "wrong"; they are different conventions. On the numbers 1 to 10 you get three different Q1 values:

import statistics

data = list(range(1, 11))
print(statistics.quantiles(data, n=4))                      # [2.75, 5.5, 8.25]
print(statistics.quantiles(data, n=4, method="inclusive"))  # [3.25, 5.5, 7.75]

A third method, often taught in school, splits the sorted data into a lower half and an upper half (leaving out the middle value when the count is odd) and takes the median of each half. For 1 to 10 that gives Q1 = 3 and Q3 = 8. With large datasets the methods agree closely; with small ones, always say which method you used.

Common mistakes

• Confusing percentage and percentile. 90 percent means 90 marks out of 100; the 90th percentile means you did at least as well as 90% of people.
• Comparing quartiles from two tools that use different methods. Fix: use one tool, or set the method explicitly.
• Deleting every flagged outlier. Fix: investigate first; keep real extreme values and document every removal.
• Using the 1.5 × IQR rule on tiny samples of 4 or 5 values, where quartiles are unstable. Fix: look at the actual values instead.