Hypothesis tests, p-values and A/B testing

Why we need tests

A food-delivery app shows a green "Order now" button to half its users and an orange one to the other half. Green gets 4.0% of users to order, orange gets 5.2%. Is orange really better, or did it just get lucky with who saw it? Random samples always differ a little. A hypothesis test tells us whether a difference is bigger than luck usually produces.

The words

• Null hypothesis (H0): the boring claim, "there is no difference". Orange and green convert equally.
• Alternative hypothesis (H1): what you suspect, "there is a difference".
• Test statistic: one number that measures how far the data is from H0, such as a z-score.
• p-value: the probability of seeing a difference at least this big IF H0 were true.
• Significance level (α, alpha): the cut-off you choose before looking, often 0.05. If p < α, you reject H0.

A two-proportion z-test, step by step

Green: 200 orders from 5,000 users (4.0%). Orange: 260 from 5,000 (5.2%). First pool the data as if H0 were true: p = 460 ÷ 10,000 = 0.046. The standard error of the difference is √(p(1 − p)(1/n_A + 1/n_B)) ≈ 0.00419. The z statistic is the difference divided by that: 0.012 ÷ 0.00419 ≈ 2.86. Then the two-sided p-value is 2 × (1 − Φ(|z|)), where Φ is the normal curve's cumulative probability.

import math
from statistics import NormalDist

ca, na, cb, nb = 200, 5000, 260, 5000
pa, pb = ca / na, cb / nb
pool = (ca + cb) / (na + nb)
se = math.sqrt(pool * (1 - pool) * (1 / na + 1 / nb))
z = (pb - pa) / se
p_value = 2 * (1 - NormalDist().cdf(abs(z)))
print(round(z, 2), round(p_value, 4))

Because 0.0042 is below 0.05, we reject H0 and conclude the orange button very likely performs better. The lift (relative improvement) is (5.2 − 4.0) ÷ 4.0 = 30%.

Reading a p-value correctly

• A p-value is NOT the probability that H0 is true.
• A p-value is NOT the probability that the result was "due to chance".
• A large p-value does NOT prove there is no difference; maybe the sample was too small to see it.
• p < 0.05 does NOT mean the effect is big or important. With crores of users, a 0.01% change can be "significant" and still not worth the work. This is statistical vs practical significance.

Two kinds of errors

Running a good A/B test

1. Decide the metric (for example, order rate) and α before starting.
2. Work out the sample size you need with a power calculation, so a real effect has a good chance (often 80%) of being detected.
3. Randomly assign users, and keep each user in one group.
4. Run for the planned time, including full weeks, since weekends behave differently.
5. Do not stop early the moment p dips below 0.05. Checking again and again ("peeking") raises false positives a lot.

Common mistakes

• Reading p = 0.03 as "97% sure the new version is better". Fix: say "if there were no difference, data this extreme would be rare (3%)".
• Stopping the test as soon as it looks good. Fix: fix the sample size in advance.
• Ignoring effect size. Fix: always report the lift and its confidence interval next to the p-value.