Sample ratio mismatch (SRM)

A chi-square goodness-of-fit check that the numbers of units assigned to each variant match the planned split; a mismatch signals a broken assignment or logging pipeline.

X^2 = \sum_{j=1}^{k} \frac{(O_j - E_j)^2}{E_j}, \qquad E_j = N\pi_j, \qquad \text{df} = k - 1

Chi-square goodness of fit against the planned allocation

$O_j$: units actually observed in variant $j$
$\pi_j$: planned share of traffic for variant $j$ (e.g. 0.5 and 0.5)
$N$: total units in the experiment
$k$: number of variants

When to use

At the start of every A/B analysis, and in A/A tests that validate the experimentation pipeline.
A strict threshold such as α = 0.001 is common practice, because the check is run on every experiment and a true SRM is usually large.

Assumptions

Units are assigned independently with the planned probabilities; counts are of the randomisation unit (users, not events).
Expected counts are large (they nearly always are).

How to interpret

A tiny p-value means the observed split is very unlikely under correct randomisation: some units were lost, duplicated or misassigned (bots filtered in one arm, a redirect that drops slow users, a logging bug).
With an SRM the arms are no longer comparable, so the treatment effect estimate is not trustworthy, however significant. Find the cause before analysing outcomes.
A non-significant SRM check does not prove the pipeline is correct; it only fails to detect an imbalance.

Common mistakes

Ignoring a "small" imbalance such as 51/49: with large N it can be overwhelming evidence of a bug.
Fixing an SRM by discarding units until the split looks right.
Checking the split on events (page views) when users were randomised.

Example

Planned 50/50, N = 10 000, observed 5 210 vs 4 790. Expected 5 000 each; $X^2 = 2 \times 210^2/5000 = 17.64$ on 1 df, p = 0.000027 < 0.001: an SRM. (5 130 vs 4 870 would give $X^2 = 6.76$ , p = 0.0093 — worth a look, but not flagged at 0.001.)