Bayesian A/B analysis: P(B > A) and expected loss

With independent Beta posteriors for two conversion rates, compute the probability that B's rate exceeds A's and the expected loss of choosing each variant.

P(\theta_B > \theta_A \given \text{data}) = \int_0^1 f_B(x)\,F_A(x)\,dx

Probability that B beats A

$\theta_A, \theta_B$: true conversion rates, with posteriors Beta( $a_A + s_A$ , $b_A + f_A$ ) and Beta( $a_B + s_B$ , $b_B + f_B$ )
$f_B, F_A$: posterior density of $heta_B$ and posterior cdf of $heta_A$

\text{Loss}(B) = \E\left[\max(\theta_A - \theta_B, 0)\right], \qquad \text{Loss}(A) = \E\left[\max(\theta_B - \theta_A, 0)\right]

Expected loss of each decision (in conversion-rate units)

$\E$: expectation over the joint posterior

When to use

Communicating an experiment's result as a direct probability ("B is better with probability 0.99") and making decisions that weigh how much could be lost by choosing wrongly.
A common decision rule: ship B when its expected loss is below a pre-set "threshold of caring".

Assumptions

Independent binomial outcomes in each arm; independent Beta priors (often Beta(1, 1)); priors and decision thresholds fixed in advance.
Randomisation and data-quality checks matter exactly as in the frequentist analysis.

How to interpret

$P(\theta_B > \theta_A)$ says nothing about how much better; pair it with the posterior of the difference or the expected loss.
Expected loss is small when either B is clearly better or the possible downside is tiny.
Repeatedly checking a Bayesian posterior does not change what the posterior says, but a stopping rule based on it still changes the procedure's long-run error rates; plan the decision rule.

Common mistakes

Treating $P(B > A) = 0.95$ as equivalent to "p = 0.05".
Ignoring the prior's influence with small samples.
Declaring a winner on probability alone when the difference is too small to matter.

Example

A: 120/2 400, B: 156/2 400, uniform priors → posteriors Beta(121, 2 281) and Beta(157, 2 245). P(B > A) = 0.987; posterior mean difference 0.0150; expected loss of choosing B = 0.00003 (0.003 percentage points), of choosing A = 0.0150. Choosing B risks very little.