Track 8 · Hypothesis Testing · Lesson 13 of 14
Multiple testing
Explain why running many tests inflates false positives, and control the family-wise error rate (Bonferroni, Holm) or the false discovery rate (Benjamini–Hochberg).
The checkout experiment found no overall effect — so an analyst sliced the data by device and by region: nine subgroups, nine tests. One of them, desktop users in the domestic market, came out with p = 0.005. Is that a discovery, or the statistical equivalent of buying enough lottery tickets?
Each test at α = 0.05 has a 5% chance of a false positive when its H0 is true. Run independent tests of true null hypotheses and the chance of at least one false positive — the family-wise error rate (FWER) — is : 23% for 5 tests, 64% for 20, 92% for 50. Look at enough subgroups, outcomes or time points and something will be “significant”.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Compute the family-wise error rate of several tests and the Bonferroni threshold
- Apply Bonferroni, Holm and Benjamini–Hochberg and interpret adjusted p-values
- Choose between FWER and FDR control for confirmatory and screening analyses
- Recognise hidden multiplicity (peeking, outcome switching, subgroup fishing) and report it honestly