Track 11 · Experiments and A/B Testing · Lesson 10 of 14
Segments and subgroup analysis
Analyse an experiment by segment responsibly: separate pre-registered from exploratory subgroups, test for differences between segments correctly, and avoid fishing for a significant slice.
The checkout redesign was inconclusive overall. Then someone splits the results by device and finds desktop users converting more with B, p = 0.054. “Let's ship B to desktop only!” Slicing until something looks good is one of the most common ways experiments mislead.
Effects can genuinely differ between groups of users — a change may help desktop users and hurt mobile ones. Looking at segments is therefore legitimate and often valuable. The danger is the garden of forking paths: there are many ways to slice the data (device, country, new vs returning, day of week, traffic source…), and if you look at enough slices, some will show a “significant” effect by chance alone.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Compute segment-level effects and test whether two segments differ
- Quantify how slicing inflates false positives and adjust for it
- Separate pre-registered (confirmatory) from exploratory segment findings
- Report segment results without over-claiming