Sample size and minimum detectable effect for an A/B test

The number of units per arm needed to detect a given difference in conversion rates with chosen α and power, and conversely the smallest effect a fixed sample can detect.

n = \frac{\left[z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} + z_{1-\beta}\sqrt{p_1(1-p_1) + p_2(1-p_2)}\right]^2}{(p_1 - p_2)^2}, \qquad \bar p = \frac{p_1 + p_2}{2}

Per-arm sample size, two-sided test, equal allocation (pooled-normal method)

$p_2$: baseline (control) conversion rate
$p_1$: treatment rate you want to be able to detect, $p_1 = p_2 + ext{MDE}$
$\alpha$: significance level (Type I error rate)
$1-\beta$: power: probability of a significant result when the true rates are $p_1$ and $p_2$
$z_{q}$: standard normal quantile at probability $q$

When to use

Before launching an experiment, to decide how long it must run (units needed ÷ eligible traffic per day).
To find the minimum detectable effect (MDE): with the traffic you can afford, the smallest true difference you would detect with the chosen power. If the MDE is larger than any plausible effect, the test is not worth running as designed.

Assumptions

Same as the planned analysis: independent units, a two-proportion z-test, a fixed sample size analysed once.
The baseline rate is a reasonable guess (from historical data); the result is sensitive to it.
With unequal allocation $n_2 = k n_1$ the engine uses $\bar p = (p_1 + k p_2)/(1 + k)$ and replaces 2 by $1 + 1/k$ and $p_2(1-p_2)$ by $p_2(1-p_2)/k$ . Results are rounded up.

How to interpret

n is per arm. Required n grows with the inverse square of the effect: halving the MDE roughly quadruples the sample.
Low baseline rates need large samples because the variance per unit is small relative to tiny absolute differences.
Power is a property of the design at an assumed effect, not of an observed result.

Common mistakes

Choosing the MDE as "the effect we expect" rather than "the smallest effect worth acting on".
Mixing absolute and relative MDEs (a 10% relative lift on 5% is 0.5 points, not 10 points).
Computing power after the test from the observed effect ("observed power"): it adds nothing beyond the p-value.
Ending early because the planned n "looks unnecessary" once results appear.

Example

Baseline 5%, target 6% (MDE = 1 percentage point), α = 0.05 two-sided, power 80%: n = 8 157.7 → 8 158 users per arm. Conversely, with 8 000 users per arm the MDE at 80% power is about 1.01 points (6.01% vs 5%).