Track 11 · Experiments and A/B Testing · Lesson 3 of 14
Choosing metrics
Choose a primary metric, secondary metrics and guardrail metrics for an experiment, define them per randomised unit, and judge the trade-off between a metric's sensitivity and its meaning.
30 minIntermediate7 graded exercisesMembers
A team tests a bolder “Buy now” button. Clicks on the button rise by 20%, but nobody checks whether purchases rose too — or whether refunds and complaints followed. What you choose to measure decides what you will conclude, so the metrics are fixed before the test starts.
An experiment should have a small, pre-declared set of metrics, each with a job:
- Primary metric (sometimes called the overall evaluation criterion): the one number the decision rests on, chosen to reflect the goal of the change. One primary metric keeps the decision honest.
- Secondary metrics: help explain why the primary moved (e.g. average order value, steps completed). They are not used to declare victory.
- Guardrail metrics: things that must not get worse — page speed, errors, refunds, unsubscribes, complaints. A change that wins on the primary but breaks a guardrail may not ship.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Assign primary, secondary and guardrail roles to metrics
- Define rates over all users assigned to each variant
- Compare metric sensitivity with the coefficient of variation
- Explain why the primary metric is fixed before the test