Statistics Quest

Experiments and A/B Testing

Design, analyse and decide from controlled experiments.

Randomisation, units and metrics, conversion rates, power and minimum detectable effects, A/A tests, analysing A/B tests, peeking, multiple variants and segments, guardrails, sequential testing foundations and decisions under uncertainty.

Lessons

  1. 1Why experiment?Explain why randomly assigning users to versions lets you read a difference in outcomes as the causal effect of the change, and why observational comparisons usually cannot.25 minFree
  2. 2Randomisation and unitsChoose the unit of randomisation for an experiment, analyse at a matching unit, and explain how correlated observations and interference distort the results.30 minMembers
  3. 3Choosing metricsChoose a primary metric, secondary metrics and guardrail metrics for an experiment, define them per randomised unit, and judge the trade-off between a metric's sensitivity and its meaning.30 minMembers
  4. 4Conversion rates and their uncertaintyEstimate a conversion rate with its standard error and a confidence interval (Wald and Wilson), and plan how many users are needed for a given precision.30 minMembers
  5. 5Sample size and minimum detectable effectCalculate the sample size an A/B test on conversion needs for a chosen power, find the minimum detectable effect for a given traffic, and explain the trade-offs between them.35 minMembers
  6. 6A/A testsUse A/A tests and sample ratio mismatch (SRM) checks to validate an experimentation pipeline before trusting any A/B result.30 minMembers
  7. 7Analysing an A/B testAnalyse a two-variant conversion test: estimate the absolute and relative lift, build a confidence interval for the difference, run the two-proportion z-test, and state an honest conclusion.40 minMembers
  8. 8Peeking and repeated looksExplain why checking a running test repeatedly and stopping at the first significant result inflates the false-positive rate, and plan a fixed-horizon test that avoids it.30 minMembers
  9. 9Multiple variantsAnalyse a test with several variants: test them jointly, control the error rate of pairwise comparisons with Bonferroni or Holm, and avoid crowning a noisy winner.40 minMembers
  10. 10Segments and subgroup analysisAnalyse an experiment by segment responsibly: separate pre-registered from exploratory subgroups, test for differences between segments correctly, and avoid fishing for a significant slice.35 minMembers
  11. 11Guardrail metricsCheck guardrail metrics for harm, frame a guardrail as a non-inferiority question with a margin, and weigh a guardrail cost against an uncertain gain.35 minMembers
  12. 12Sequential testing foundationsExplain how group-sequential designs allow planned interim looks without inflating the false-positive rate, using the idea of an alpha-spending function, and compare them with fixed-horizon tests.30 minMembers
  13. 13Deciding under uncertaintyTurn an experiment result into a business decision by comparing the confidence interval with the smallest effect worth acting on, estimating the expected value of shipping, and stating the remaining uncertainty honestly.35 minMembers
  14. 14Common experimentation mistakesRecognise and fix the most common ways A/B tests go wrong — novelty effects, sample ratio mismatch, wrong units, peeking, fishing, and Simpson's paradox from changing allocations — and audit an experiment report.35 minMembers