Checkout redesign A/B test

A 14-day online experiment at a fictional e-commerce site, Nakhla Market. About 16 000 new visitors were randomly assigned to the current checkout (A) or a redesigned one-page checkout (B). Each row is one user with the variant, device, country segment, signup day, whether they bought (converted), revenue and the page-load time of the checkout page as a guardrail metric. Built for two-proportion tests, confidence intervals for a lift, revenue comparisons with skewed data, guardrails and the dangers of peeking at daily results.

Open in the lab Download CSV

Rows: 16,000
Columns: 9
One row is: one user in the experiment
Version: v1

Data dictionary

Column	Type	Unit	Summary	Description
`user_id` User	id			Anonymous user identifier.
`signup_day` Test day	integer	day	mean 7.06, median 7, SD 4.054, range 1–14	Day of the test on which the user entered the experiment (1–14).
`signup_date` Signup date	date			Calendar date the user entered the experiment.
`variant` Variant	category		2 levels: A, B	A = current checkout (control), B = redesigned checkout (treatment).
`device` Device	category		3 levels: mobile, desktop, tablet	Device type.
`segment` Country segment	category		3 levels: domestic, gcc, international	Visitor location segment.
`converted` Converted	integer		mean 0.03075, median 0, SD 0.1726, range 0–1	1 if the user completed a purchase within the test, else 0 (coded 0/1 so means are conversion rates).
`revenue_sar` Revenue	number	SAR	mean 6.968, median 0, SD 51.08, range 0–2094	Order value; 0 for users who did not convert.
`page_load_ms` Checkout load time	integer	ms	mean 1622, median 1551, SD 514, range 395–5042	Load time of the checkout page for the user's first visit (guardrail metric).

What it is good for

Estimate conversion rates per variant and a confidence interval for the absolute and relative lift.
Run a two-proportion z-test or chi-square test and interpret it in business terms.
Analyse revenue per user when most values are zero and the rest are skewed.
Check a guardrail metric (page-load time) and a sample-ratio mismatch.
See how cumulative daily p-values fluctuate and why peeking inflates false positives.

Methods: two-proportion z-test, chi-square test, confidence interval for a difference in proportions, Welch t-test, Mann–Whitney test, sample ratio mismatch check, sequential peeking.

Data quality notes

True conversion rates are 3.1% (A) and 3.5% (B); observed rates differ slightly because of sampling variation.
revenue_sar is 0 for about 97% of users and heavily right-skewed among converters.
Variant B's checkout is about 6% slower — a guardrail trade-off to discuss.
The variant split is not exactly 50/50 because assignment was random per user; it is consistent with a fair coin.
No missing values.

Provenance and licence

Each user is assigned A or B with probability 0.5 (so the split is close to, but not exactly, 50/50). Device (mobile 62%, desktop 33%, tablet 5%) and segment are drawn independently of the variant. True conversion probability is 3.1% for A and 3.5% for B, multiplied by a device factor (desktop 1.3, mobile 0.85, tablet 1.0, averaging ≈1). Revenue for converters is log-normal (median ≈180 SAR) and identical in distribution for both variants. Page-load time is log-normal (median ≈1.35 s on desktop, slower on mobile) and 6% slower under B.

Synthetic data · CC0-1.0 (synthetic data generated by Statistics Quest) · generated deterministically (seed 20241005). It does not describe real people or organisations; any resemblance is coincidental.

Preview

Dataset

Checkout redesign A/B test

16,000 rows · one row = one user in the experiment · synthetic data

About this dataset Open in the lab

User	Test day (day)	Signup date	Variant	Device	Country segment	Checkout load time (ms)
U00001	1	2024-05-05	B	mobile	domestic	1781
U00002	1	2024-05-05	B	mobile	domestic	1842
U00003	1	2024-05-05	B	mobile	domestic	1197
U00004	1	2024-05-05	A	desktop	gcc	1035
U00005	1	2024-05-05	B	tablet	gcc	1774
U00006	1	2024-05-05	A	desktop	domestic	1609
U00007	1	2024-05-05	A	mobile	domestic	899
U00008	1	2024-05-05	B	tablet	domestic	2025
U00009	1	2024-05-05	A	mobile	domestic	1837
U00010	1	2024-05-05	A	mobile	domestic	1485
U00011	1	2024-05-05	B	desktop	domestic	1251
U00012	1	2024-05-05	A	desktop	domestic	1886

Used in

Interval for a difference in proportions · Estimation and Confidence Intervals
Tests for a proportion · Hypothesis Testing
Comparing two proportions · Hypothesis Testing
Multiple testing · Hypothesis Testing
Reporting a test responsibly · Hypothesis Testing
Why experiment? · Experiments and A/B Testing
Choosing metrics · Experiments and A/B Testing
Conversion rates and their uncertainty · Experiments and A/B Testing
Sample size and minimum detectable effect · Experiments and A/B Testing
A/A tests · Experiments and A/B Testing
Analysing an A/B test · Experiments and A/B Testing
Peeking and repeated looks · Experiments and A/B Testing
Segments and subgroup analysis · Experiments and A/B Testing
Guardrail metrics · Experiments and A/B Testing
Deciding under uncertainty · Experiments and A/B Testing
Common experimentation mistakes · Experiments and A/B Testing
Beta–binomial updating · Advanced Foundations
Bayesian A/B testing · Advanced Foundations
Customer conversion experiment: ship the new checkout? · Capstone