Track 11 · Experiments and A/B Testing · Lesson 8 of 14
Peeking and repeated looks
Explain why checking a running test repeatedly and stopping at the first significant result inflates the false-positive rate, and plan a fixed-horizon test that avoids it.
On day three of a two-week test the dashboard turns green: p = 0.03. The product manager wants to ship today — “it's significant!”. If the team stops whenever the dashboard turns green, they will ship far more duds than their 5% significance level suggests.
A significance level α = 0.05 promises: if there is no real difference, this test will wrongly declare one 5% of the time. That promise is about one analysis at a sample size fixed in advance.
Peeking means looking at the p-value repeatedly while data accumulate and stopping as soon as it drops below α. Each look is another chance for noise to cross the line. The p-value of a test with no real effect wanders up and down; given enough looks, it will dip below 0.05 at some point far more often than 5% of the time.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Explain why stopping at the first significant look inflates false positives
- Compute cumulative interim results and recognise their instability
- Convert a sample size into a fixed test duration in whole weeks
- Separate legitimate monitoring from optional stopping