Track 11 · Experiments and A/B Testing · Lesson 9 of 14
Multiple variants
Analyse a test with several variants: test them jointly, control the error rate of pairwise comparisons with Bonferroni or Holm, and avoid crowning a noisy winner.
A language-learning app tests four monthly prices at once. One price shows the highest revenue per visitor, and the team is ready to switch. But with four arms there are six pairwise comparisons, and the arm that looks best is often just the luckiest.
Testing several variants against each other is efficient — one experiment answers several questions — but every extra comparison is another chance for a false positive. With variants there are pairs. If each is tested at α = 0.05, the family-wise error rate (FWER), the probability of at least one false positive when no variant truly differs, grows quickly.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Count comparisons and compute the family-wise error rate
- Apply Bonferroni and Holm corrections
- Start with a joint test before pairwise comparisons
- Recognise the winner's curse when several arms are close