Track 11 · Experiments and A/B Testing · Lesson 13 of 14
Deciding under uncertainty
Turn an experiment result into a business decision by comparing the confidence interval with the smallest effect worth acting on, estimating the expected value of shipping, and stating the remaining uncertainty honestly.
A test is not a verdict machine. A statistically significant lift can be too small to pay for itself, and a non-significant one can still be worth shipping when it is cheap and the downside is small. Good experimenters translate intervals into money, risk and a clear recommendation.
Statistical significance answers: is the effect distinguishable from zero? Practical significance answers: is it big enough to matter? Decide the minimum worthwhile effect (MWE) before the test — the lift that would pay for building, maintaining and risking the change — and then read the confidence interval against both 0 and the MWE.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Read a confidence interval against zero and a minimum worthwhile effect
- Estimate the expected value and break-even lift of shipping
- Use P(B > A) and expected loss as decision aids, knowing what they omit
- Write a recommendation that weighs evidence, value, costs and uncertainty