Track 13 · Resampling and Statistical Reasoning · Lesson 2 of 10
Permutation tests
Run and interpret a permutation test by shuffling group labels, and distinguish exact from Monte Carlo permutation p-values.
A courier's new routing system seems to save about 3 minutes per delivery, but individual deliveries vary by 10 minutes or more. Could a 3-minute gap appear just from how the deliveries happened to fall into the two groups? A permutation test answers that by literally reshuffling the groups.
A permutation test builds the null distribution from the data instead of from a formula. Its null hypothesis is that the group labels do not matter: the outcome of each unit would have been the same whichever group it was in. Statisticians call this exchangeability — under the null hypothesis the labels are interchangeable.
If the labels do not matter, then every way of re-dealing them is equally plausible. So we re-deal them many times, recompute the statistic each time, and see where the observed value falls among the shuffled ones.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Explain the null hypothesis of a permutation test as exchangeable labels
- Compute an exact permutation p-value by enumerating splits or sign patterns
- Report Monte Carlo permutation p-values with the (count + 1)/(B + 1) rule, the seed and B
- Choose a shuffling scheme that matches the design: labels, signs within pairs, or whole clusters