Track 11 · Experiments and A/B Testing · Lesson 14 of 14
Common experimentation mistakes
Recognise and fix the most common ways A/B tests go wrong — novelty effects, sample ratio mismatch, wrong units, peeking, fishing, and Simpson's paradox from changing allocations — and audit an experiment report.
35 minAdvanced7 graded exercisesMembers
Most failed experiments do not fail because of exotic statistics. They fail because of a handful of recurring mistakes that an experienced reviewer spots in minutes. This lesson is that reviewer's checklist.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Run a reviewer's checklist on any experiment report
- Check for novelty effects by comparing entry cohorts
- Explain how a mid-test allocation change can produce Simpson's paradox
- Name the consequence and the remedy for each common mistake