Track 11 · Experiments and A/B Testing · Lesson 5 of 14
Sample size and minimum detectable effect
Calculate the sample size an A/B test on conversion needs for a chosen power, find the minimum detectable effect for a given traffic, and explain the trade-offs between them.
A team plans to test a checkout redesign for two weeks because “two weeks is what we always do”. If the effect they hope for needs four times their traffic to detect, the test is doomed to an inconclusive result before it starts. Sample size planning is where good experiments are won.
Power is the probability that a test detects an effect (rejects H0) when the effect really has a given size. Four quantities are linked, and fixing any three determines the fourth:
- the baseline rate and the effect you want to detect (the target );
- the significance level (usually 0.05, two-sided);
- the power (usually 80% or 90%);
- the sample size per group .
The minimum detectable effect (MDE) is the smallest effect that a test of a given size detects with the chosen power. It should be the smallest effect that would be worth acting on — not the effect you secretly hope for.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Calculate the per-group sample size for a two-proportion test with the classic formula
- Find the minimum detectable effect for the traffic available
- Distinguish absolute and relative lifts when planning
- Explain why an underpowered test cannot show that an effect is absent