Track 11 · Experiments and A/B Testing · Lesson 1 of 14
Why experiment?
Explain why randomly assigning users to versions lets you read a difference in outcomes as the causal effect of the change, and why observational comparisons usually cannot.
A product team notices that shoppers who use the new “save for later” button buy three times as often as those who do not. Should the company make the button bigger, or are keen shoppers simply the ones who click it? Only an experiment can tell the two stories apart.
Product and business teams constantly ask causal questions: Does the redesigned checkout make people buy more? Does a lower price bring in enough extra subscribers to pay for itself? A causal question is about what would happen if we changed something, not just about which things go together.
Data collected by watching what people do anyway — observational data — answers a different question: who tends to do what. The two are easily confused.
| Group | Visitors | Purchasers | Conversion rate |
|---|---|---|---|
| Used “save for later” | 2,000 | 180 | 9.0% |
| Did not use it | 18,000 | 540 | 3.0% |
Users of the button convert at 9.0% against 3.0% — 3.0 times as often. But the users chose to click. People who were already planning to buy, who visit often, or who have full baskets are far more likely to save items. Their high conversion rate could be entirely due to who they are, not to the button. This is self-selection, a form of confounding: a third factor (purchase intent) drives both the behaviour and the outcome.
- a user's outcome (e.g. 1 = bought) with the change and without it — the two potential outcomes. We only ever see one of them for each user.
- the average over all users
- the observed average outcome in the treatment group B and in the control group A
We can never see the same user both with and without the change. A randomised controlled experiment (an A/B test in product work) solves this by letting a coin decide who gets which version. Because the coin knows nothing about the user, the two groups are alike on average in everything — intent, device, mood, wealth, things we have measured and things we have not. Any difference in outcomes beyond chance variation can then be attributed to the change itself.
Worked example
Did the coin balance the groups?
In the (synthetic) checkout_ab experiment, 16 000 new visitors were randomly assigned to the current checkout (A) or a redesign (B). Device and location were recorded before assignment could affect anything. Are they balanced?
Device. Mobile users make up 62.1% of group A and 61.4% of group B.
Location. Domestic visitors are 69.8% of A and 69.9% of B.
Read it. The shares differ by well under one percentage point — the size of difference chance alone produces with 8 000 users per group. A balance table like this is a check that the randomisation was carried out correctly (no bug sent, say, more desktop users to B). It cannot show that unmeasured factors are balanced — that guarantee comes from the coin itself, which balances measured and unmeasured factors alike on average.
The groups are comparable on pre-treatment characteristics, so a difference in conversion can be read causally (allowing for chance).
Dataset
Checkout redesign A/B test
16,000 rows · one row = one user in the experiment · synthetic data
| User | Test day (day) | Signup date | Variant | Device | Country segment | Converted | Revenue (SAR) | Checkout load time (ms) |
|---|---|---|---|---|---|---|---|---|
| U00001 | 1 | 2024-05-05 | B | mobile | domestic | 0 | 0 | 1781 |
| U00002 | 1 | 2024-05-05 | B | mobile | domestic | 0 | 0 | 1842 |
| U00003 | 1 | 2024-05-05 | B | mobile | domestic | 0 | 0 | 1197 |
| U00004 | 1 | 2024-05-05 | A | desktop | gcc | 0 | 0 | 1035 |
| U00005 | 1 | 2024-05-05 | B | tablet | gcc | 0 | 0 | 1774 |
| U00006 | 1 | 2024-05-05 | A | desktop | domestic | 0 | 0 | 1609 |
One row per user: assigned variant, device, location segment, signup day, whether they bought, revenue and checkout page-load time.
Quick check
Why does random assignment allow a causal reading of the difference between groups?
The life of an experiment (what this track covers)
- Write the hypothesis and the decision it will inform.
- Choose the unit of randomisation and the metrics (primary, secondary, guardrails).
- Plan the sample size and duration from the minimum effect worth detecting.
- Randomise, run, and check the plumbing (A/A tests, sample ratio mismatch).
- Analyse once, at the planned end: difference, confidence interval, test.
- Decide, weighing the uncertainty, the guardrails and the business costs.
Exercises
The lesson is complete when every exercise is answered correctly. Hints are free; viewing a worked solution before solving means the exercise won't count towards mastery.
0 of 6 exercises correct (not saved — sign in to keep your progress)
Exercise 1 · Interpretation, difficulty 1 of 3
Why the button looks magical
Users of the “save for later” button converted at 9.0%, non-users at 3.0%. What is the most important reason this comparison cannot tell us the effect of the button?
Exercise 2 · Practice, difficulty 1 of 3
Experiment or observation?
Classify each study as a randomised experiment or an observational study.
Exercise 3 · Application, difficulty 2 of 3
Check the balance
In checkout_ab, randomisation should balance pre-treatment characteristics. Calculate the percentage of users on desktop within variant A and within variant B (2 decimal places).
Data: checkout_ab · Open in the lab to explore and calculate.
Exercise 4 · Challenge, difficulty 3 of 3
A feature that does nothing
A wish-list feature was used by some visitors but not others. Visitors fall into two types:
| Type | Used the feature | Did not | Conversion (both) |
|---|---|---|---|
| Engaged | 800 | 1700 | 10% |
| Casual | 500 | 6500 | 1% |
Within each type, users and non-users convert at exactly the same rate. Calculate the overall conversion rate of feature users and of non-users (%, 2 decimals), and the naive difference users − non-users (percentage points, 2 decimals).
Exercise 5 · Practice, difficulty 1 of 3
Order the experiment
Put the steps of a well-run A/B test in order.
Exercise 6 · Interpretation, difficulty 2 of 3
Answer the product manager
A product manager writes: “Save-for-later users convert at 9.0% versus 3.0%. Let's make the button twice as big — it clearly drives purchases.” In two to four sentences, explain why the data do not show that, and propose how to find out.
What you can now do
- Distinguish a causal question from an association
- Explain self-selection and confounding in product data
- Explain why random assignment supports causal conclusions
- Check covariate balance in a randomised experiment