Analysing an A/B test on conversion rates

Estimate each variant's conversion rate with a Wilson interval, then estimate and test the difference and report the relative lift with its uncertainty.

\hat p = \frac{x}{n}, \qquad \frac{\hat p + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}

Conversion rate and its Wilson score interval

$x$: number of converting units (users, sessions) in the variant
$n$: number of units randomised to the variant
$z$: standard normal critical value, $z_{0.975} \approx 1.96$ for 95%

(\hat p_B - \hat p_A) \pm z\sqrt{\frac{\hat p_A(1-\hat p_A)}{n_A} + \frac{\hat p_B(1-\hat p_B)}{n_B}}

Interval for the difference (unpooled standard error)

$\hat p_A, \hat p_B$: observed conversion rates of control (A) and treatment (B)
$n_A, n_B$: units in each arm

z = \frac{\hat p_B - \hat p_A}{\sqrt{\bar p(1-\bar p)\left(\frac{1}{n_A} + \frac{1}{n_B}\right)}}, \qquad \bar p = \frac{x_A + x_B}{n_A + n_B}

Two-proportion z-test (pooled standard error under H0)

$\bar p$: pooled conversion rate, the common rate assumed by the null hypothesis
$x_A, x_B$: conversions in each arm

\text{relative lift} = \frac{\hat p_B - \hat p_A}{\hat p_A}

Relative lift

$\hat p_A$: baseline (control) conversion rate

When to use

A randomised experiment with a binary outcome per unit (converted or not) and two variants.
Report the absolute difference in percentage points as the primary result, with its interval; add the relative lift for communication.

Assumptions

Units were randomised and analysed at the same level (the unit of randomisation is the unit of analysis).
Outcomes of different units are independent (no interference, no repeated sessions counted as separate units when users were randomised).
The sample size and stopping rule were fixed in advance (see peeking) and the split matches the plan (see sample ratio mismatch).
Normal approximations need enough conversions and non-conversions in each arm (roughly 10 or more of each).

How to interpret

The interval for $p_B - p_A$ gives the effect sizes compatible with the data; check it against the smallest effect worth acting on, not just against 0.
The p-value is the probability, if the variants truly converted at the same rate, of a difference at least as extreme as the one observed. It is not the probability that B is no better.
The engine's z-test uses the pooled SE (the null assumes one common rate) while the interval uses the unpooled SE, so in borderline cases the two can disagree slightly. Newcombe's hybrid-score interval is a better-behaved alternative for small counts.

Common mistakes

Quoting only the relative lift ("+30%!") without the baseline or the interval: a 30% lift on a 0.1% rate is tiny in absolute terms.
Counting page views when users were randomised (inflates n and understates uncertainty).
Stopping as soon as p < 0.05, testing many metrics or segments and reporting the best one.
Treating "not significant" as "no effect": a wide interval that includes useful effects is inconclusive.

Example

A: 120 conversions out of 2 400 (5.0%, Wilson 95% CI 4.20% to 5.95%). B: 156 out of 2 400 (6.5%, Wilson 95% CI 5.58% to 7.56%).

Difference B − A = 1.5 percentage points, 95% CI 0.18 to 2.82 points (unpooled Wald; Newcombe gives 0.18 to 2.83).
Pooled z = 2.23, two-sided p = 0.026.
Relative lift = 0.015/0.05 = 30%, but the interval shows the true lift could be anywhere from about 4% to 56% of the baseline rate: the direction is supported, the size is uncertain.