Track 11 · Experiments and A/B Testing · Lesson 11 of 14
Guardrail metrics
Check guardrail metrics for harm, frame a guardrail as a non-inferiority question with a margin, and weigh a guardrail cost against an uncertain gain.
A redesigned checkout might lift sales — but if it also makes the page noticeably slower, it can hurt every other part of the site, now and in future tests. Guardrails are the metrics that stop a team optimising one number at the expense of everything else.
A guardrail metric measures something that must not get (much) worse: speed, errors, crashes, refunds, complaints, unsubscribes, or a business metric owned by another team. Guardrails are declared before the test, each with a tolerance: how much deterioration is acceptable.
The question for a guardrail is not “is there a significant difference?” but “can we rule out unacceptable harm?”. That reverses the usual logic.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Declare guardrails with tolerances before a test
- Check a guardrail as non-inferiority using a one-sided upper bound
- Look at tails (percentiles) as well as means
- Weigh a clear guardrail cost against an uncertain primary gain