Track 11 · Experiments and A/B Testing · Lesson 12 of 14
Sequential testing foundations
Explain how group-sequential designs allow planned interim looks without inflating the false-positive rate, using the idea of an alpha-spending function, and compare them with fixed-horizon tests.
Peeking is dangerous, but waiting four weeks to stop an obviously harmful or obviously excellent change is costly too. Sequential designs give a disciplined middle way: decide in advance when you will look and how much of your error budget each look may spend.
A fixed-horizon test spends its entire false-positive budget α at one moment: the planned end. Naive peeking spends α = 0.05 at every look, so the total climbs (about 14% with 5 looks and 20% with 10, by simulation — see peeking).
A group-sequential design fixes the looks in advance (for example after 20%, 40%, … of the planned users) and uses stricter thresholds at each look, chosen so that the probability of ever crossing a threshold when H0 is true is exactly α overall.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Explain how planned interim looks with stricter thresholds control the overall α
- Evaluate O'Brien–Fleming-type and Pocock-type spending functions
- Use a Bonferroni split as a simple conservative sequential rule
- Recognise that estimates after early stopping tend to be inflated