Track 13 · Resampling and Statistical Reasoning · Lesson 9 of 10
p-hacking and publication bias
Explain how multiple testing, flexible analysis and selective publication produce false or exaggerated findings, and apply corrections and safeguards such as adjustment and pre-registration.
Search three funds' daily returns for a “day-of-the-week effect” and you will find one: 4 of 15 weekday comparisons in a dataset built with no such effect have p < 0.05. Publish only those, and the literature fills with discoveries that are noise.
A test at promises that, when there is no effect, it will wrongly declare one only 5% of the time. That promise is about one pre-specified test. It breaks when:
- Many tests are run and only the significant ones are reported (multiple testing).
- The analysis is chosen after seeing the data: which outcome, which subgroup, which outliers to drop, which covariates, when to stop collecting — the garden of forking paths. The analyst may run only one test, but would have run a different one had the data looked different.
- p-hacking: deliberately trying variations until p drops below 0.05.
- Publication bias: journals, managers and researchers favour significant results, so non-significant studies stay in the “file drawer”.
Included with a Formula Quest membership
It has 8 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Compute the family-wise error rate and apply Bonferroni, Holm and Benjamini–Hochberg adjustments
- Recognise forking paths, optional stopping and HARKing as sources of false positives
- Explain how publication bias exaggerates effects in underpowered literatures
- Use pre-registration, complete reporting and replication as safeguards