Statistics Quest

Confidence interval for a difference in proportions

Interval for p₁ − p₂ from two independent samples, using the unpooled (Wald) standard error or Newcombe's hybrid score method.

(p^1−p^2)±z1−α/2p^1(1−p^1)n1+p^2(1−p^2)n2(\phat_1 - \phat_2) \pm z_{1-\alpha/2}\sqrt{\frac{\phat_1(1-\phat_1)}{n_1} + \frac{\phat_2(1-\phat_2)}{n_2}}
Unpooled (Wald) interval
p^i=xi/ni\phat_i = x_i/n_i
sample proportion in group ii
nin_i
size of group ii
[ d−(p^1−l1)2+(u2−p^2)2,    d+(u1−p^1)2+(p^2−l2)2 ]\Big[\,d - \sqrt{(\phat_1 - l_1)^2 + (u_2 - \phat_2)^2},\;\; d + \sqrt{(u_1 - \phat_1)^2 + (\phat_2 - l_2)^2}\,\Big]
Newcombe hybrid score interval (better for small counts)
dd
p^1−p^2\phat_1 - \phat_2
(li,ui)(l_i, u_i)
Wilson score interval for group ii

When to use

Comparing two rates: conversion in variant B vs A, default rates in two segments, recovery under two treatments. Newcombe's interval is preferred when counts are small or rates are near 0 or 1; the Wald interval is fine for large samples.

Assumptions

Two independent random samples (or randomised groups); independent trials within each group.

How to interpret

  • The interval is for the absolute difference in percentage points. A relative lift (p^B/p^A−1\phat_B/\phat_A - 1) is a different quantity.
  • The test of p1=p2p_1 = p_2 uses a pooled SE, while the interval uses the unpooled SE, so in borderline cases the test and the Wald interval can disagree slightly.

Common mistakes

  • Using the pooled SE for the interval.
  • Confusing percentage points with percent change.
  • Comparing two separate intervals instead of computing one for the difference.

Example

Variant B: 48 of 400 (12.0%); A: 30 of 400 (7.5%). Difference 0.0450.045. Wald 95% interval: (0.0040,0.0860)(0.0040, 0.0860); Newcombe: (0.0037,0.0868)(0.0037, 0.0868). Both suggest an improvement between about 0.4 and 8.7 percentage points.