Track 11 · Experiments and A/B Testing · Lesson 2 of 14
Randomisation and units
Choose the unit of randomisation for an experiment, analyse at a matching unit, and explain how correlated observations and interference distort the results.
An analyst randomises users to two checkouts but counts every session as an independent observation. The p-value comes out at 0.03 and the change ships — yet the evidence was much weaker than it looked. Getting the unit right is the first thing an experienced reviewer checks.
Every experiment has a unit of randomisation: the thing the coin is flipped for. Common choices are the user (or account, or device), the session, the page view, or a whole cluster such as a store, a city or a time window.
It also has a unit of analysis: the thing each row of your analysis represents. The safe rule is simple: analyse at the unit you randomised (or use a method that accounts for the grouping).
Included with a Formula Quest membership
It has 8 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Choose user, session or cluster randomisation for a given change
- Calculate a design effect and an effective sample size
- Explain why analysing below the randomisation unit understates variance
- Recognise interference and when to randomise clusters
- Use blocking to balance a strong known predictor when units are few