Track 16 · Advanced Foundations · Lesson 4 of 8
Bayesian A/B testing
Analyse an A/B test with Beta posteriors — P(B > A), the posterior of the difference and expected loss — and make a decision while respecting the method's assumptions and limits.
A product team does not want a p-value; it wants to know “how likely is it that the new checkout is better, and what do we risk if we ship it and we're wrong?” Bayesian A/B analysis answers both questions directly — provided its prior and model are taken as seriously as any other assumption.
Give each variant's conversion rate its own Beta prior and update it with that variant's data. The two posteriors, and , describe what we know about each rate. Every question about the comparison becomes a question about these two distributions.
Included with a Formula Quest membership
It has 8 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Build Beta posteriors for two variants and compute P(B > A)
- Use expected loss and a pre-agreed threshold to make a decision
- Check how a sceptical prior changes a small-sample conclusion
- Explain why Bayesian analysis does not remove the need for randomisation, planning or caution with peeking
- Treat unplanned subgroup contrasts as hypotheses, not findings