Statistics Quest

Track 11 · Experiments and A/B Testing · Lesson 7 of 14

Analysing an A/B test

Analyse a two-variant conversion test: estimate the absolute and relative lift, build a confidence interval for the difference, run the two-proportion z-test, and state an honest conclusion.

40 minAdvanced7 graded exercisesMembers

The checkout redesign test has finished: 16 000 users, two weeks, one primary metric. The product lead wants a one-line answer — ship it or not? Your job is to turn the counts into an estimate, an interval and a test, and to say exactly what they do and do not show.

An A/B analysis on a conversion metric has four parts, always reported together:

  1. the two rates p^A\hat p_A and p^B\hat p_B;
  2. the effect: the absolute difference p^B−p^A\hat p_B - \hat p_A (in percentage points) and the relative lift (p^B−p^A)/p^A(\hat p_B - \hat p_A)/\hat p_A (in %);
  3. a confidence interval for the difference;
  4. a test of H0: pB=pAp_B = p_A (two-sided, unless a one-sided test was pre-registered).

Included with a Formula Quest membership

It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:

  • Compute absolute and relative lifts and say which is which
  • Run a two-proportion z-test with the pooled standard error
  • Build Wald and Newcombe intervals for a difference in rates
  • State an inconclusive result honestly, without claiming no effect
View Formula Quest membershipsBrowse free lessons