Peeking and sequential testing

Checking a fixed-sample test repeatedly and stopping at the first significant result inflates the false-positive rate well above α; sequential designs spend α deliberately across planned looks.

P(\text{reject at some look} \given H_0) = P\left(\bigcup_{k=1}^{K} \{|Z_k| > z_{1-\alpha/2}\}\right) > \alpha \quad (K > 1)

Why peeking inflates the Type I error

$K$: number of looks at the accumulating data
$Z_k$: test statistic computed with the data available at look $k$

\alpha(t) = 2 - 2\,\Phi\!\left(\frac{z_{1-\alpha/2}}{\sqrt{t}}\right), \quad 0 < t \le 1

An O'Brien–Fleming-type alpha-spending function (Lan–DeMets)

$t$: information fraction: share of the planned sample observed so far
$\alpha(t)$: cumulative Type I error allowed to be spent by information fraction $t$ ; $alpha(1) = alpha$
$\Phi$: standard normal cdf

When to use

Whenever results are visible while an experiment runs and someone may want to stop early.
Choose either a fixed-horizon design (decide n in advance, analyse once) or a sequential design that plans the looks and adjusts the critical values.

Assumptions

Under H0 the z statistics at successive looks are strongly correlated (each look reuses earlier data), but every extra look is another chance to cross the line by luck.
Sequential boundaries are valid only if the looks and the spending rule are fixed before the data arrive.

How to interpret

A p-value from a test that was stopped "when it became significant" does not have its nominal meaning: the procedure's false-positive rate is larger than α.
Alpha-spending designs (Pocock, O'Brien–Fleming, Lan–DeMets) make early stopping legitimate by demanding stronger evidence at early looks; O'Brien–Fleming-type spending keeps the final critical value close to the fixed-sample one.
Stopping early for a large effect also tends to overestimate the effect size.

Common mistakes

"We only looked, we did not act": looking with the option to stop is what inflates the error rate.
Extending a test that is "almost significant" (adding data until it crosses) — the same problem in reverse.
Applying a fixed-sample p-value to a sequential design, or making up boundaries after seeing the data.

Example

A simulation with the engine's seeded generator (seed 2026, 20 000 A/A experiments, no true difference, normal outcomes, 10 looks after every 100 users per arm, two-sided z at α = 0.05 each look): rejecting at the final look only gave a false-positive rate of 4.8%; stopping at the first significant look gave 19.3% — about four times the nominal 5%.