Track 8 · Hypothesis Testing · Lesson 9 of 14

Comparing two proportions

Compare two independent proportions with the two-proportion z-test, estimate the difference with a confidence interval, and interpret an A/B test result in business terms.

35 minIntermediate7 graded exercisesMembers

An e-commerce team ran a 14-day experiment: 16 000 visitors were randomly shown the current checkout (A) or a redesigned one (B). B converted slightly better. Should the company roll it out? The two-proportion test is the workhorse of A/B testing — and this experiment shows why its interval matters more than its p-value.

Two independent groups, a yes/no outcome: the parameter is the difference of population proportions $p_1 - p_2$ . Under $H_0: p_1 = p_2$ both groups share one common proportion, so the test estimates it by pooling the groups.

Included with a Formula Quest membership

It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:

Run a two-proportion z-test with the pooled null standard error
Estimate a difference in proportions with an unpooled (or Newcombe) confidence interval
Distinguish percentage points from relative change
Read an inconclusive A/B test from its interval rather than its p-value

View Formula Quest memberships Browse free lessons

Formula sheet

The two-proportion z-test uses the pooled proportion (because H0 says the proportions are equal)
$z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\bar{p}(1 - \bar{p})\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}}, \qquad \bar{p} = \frac{x_1 + x_2}{n_1 + n_2}$
The (Wald) confidence interval for the difference uses the unpooled standard error — no H0 is assumed when estimating
$(\hat{p}_1 - \hat{p}_2) \pm z_{1-\alpha/2}\sqrt{\frac{\hat{p}_1(1 - \hat{p}_1)}{n_1} + \frac{\hat{p}_2(1 - \hat{p}_2)}{n_2}}$

Reference

Confidence interval for a difference in proportions
Interval for p₁ − p₂ from two independent samples, using the unpooled (Wald) standard error or Newcombe's hybrid score method.
Two-proportion z-test
Compares two independent proportions, using the pooled proportion for the test and the unpooled standard error for the confidence interval.
Analysing an A/B test on conversion rates
Estimate each variant's conversion rate with a Wilson interval, then estimate and test the difference and report the relative lift with its uncertainty.

Working method

Read the question twice: what is being asked, about which group or variable?
Name the method before calculating.
Keep full precision; round only the final answer.
Ask whether the answer is plausible in context.