Lab · Data Science, Machine Learning and AI
Lab: A/B test of a UPI Pay button
The brief
A payments app tried a new green "Pay now" button on half its users for a week. Add up the results, work out conversion rates and lift, run a two-proportion z-test and decide whether to ship the new button.
Why it matters
Product teams at payments, shopping and food apps run A/B tests every week before changing a screen. The analyst's job is to say whether a difference is real or just luck, and to stop teams shipping changes on the strength of one lucky day.
Skills you practise
- conversion rates
- relative lift
- two-proportion z-test
- p-values with NormalDist
- making a decision with a significance level
What you will have at the end
abtest.py: a reusable A/B test checker, plus a three-sentence note to the product manager with the rates, the lift, the p-value and the decision.
How it is marked (rubric, 100 points)
| Correct testAll four checks pass: totals, rates, lift, z, p-value and decision. | 50 |
|---|---|
| Careful decisionYou explain why day 1 alone was not enough, and you report percentage points and lift separately. | 25 |
| Readable codeThe formulas are easy to follow, with no repeated code, and z is reused from step 3. | 15 |
| Clear noteThe note to the product manager is short, plain and gives the numbers that matter. | 10 |
0/4 steps done
Step 1 of 4
Add up the week
totals(DAILY_CSV) returns {"A": (5000, 200), "B": (5000, 260)}: visitors and payments per variant over all seven days.
The file has one row per day per variant. Read it with csv.DictReader, convert visitors and payments to int, and keep a running (visitors, payments) pair for each variant. out.get(variant, (0, 0)) gives the pair so far, or zeros the first time you see a variant.
Run shows what your code prints. Check runs the step's tests. Ctrl+Enter checks too.