Track 11 · Experiments and A/B Testing · Lesson 6 of 14
A/A tests
Use A/A tests and sample ratio mismatch (SRM) checks to validate an experimentation pipeline before trusting any A/B result.
A team is thrilled: their new feature lifts conversion by 4% with p = 0.002. Then someone notices the treatment group has 3% fewer users than planned — the new page was dropping slow-loading visitors before they were counted. The “win” was a measurement artefact. Checking the plumbing comes before reading the result.
An A/A test splits users exactly as an A/B test would, but both groups see the same version. Nothing differs, so any “effect” is chance or a bug. A/A tests check that:
- assignment is random and splits traffic in the planned ratio;
- logging, joins and metric definitions are identical across groups;
- the analysis produces false positives at the advertised rate α — no more.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Explain what an A/A test validates
- Run a chi-square SRM check for equal and unequal planned splits
- Recognise likely causes of sample ratio mismatch
- Interpret the A/A false-positive rate as a check on the analysis