Guardrail metrics

Metrics that must not get meaningfully worse (errors, latency, complaints, refunds) — judged by whether the interval rules out harm larger than a pre-agreed tolerance, not by a non-significant p-value.

\text{acceptable if } U_{1-\alpha} < \delta, \qquad U_{1-\alpha} = (\hat p_B - \hat p_A) + z_{1-\alpha}\sqrt{\frac{\hat p_A(1-\hat p_A)}{n_A} + \frac{\hat p_B(1-\hat p_B)}{n_B}}

Non-inferiority style check for a 'lower is better' rate

$U_{1-\alpha}$: one-sided upper confidence bound for the increase $p_B - p_A$
$\delta$: largest increase in the harmful rate the business is willing to tolerate (set before the test)
$\hat p_A, \hat p_B$: observed rates of the harmful event in control and treatment

When to use

Every A/B test: alongside the primary (decision) metric, list a few guardrails that protect users and the business — error rates, page load time, cancellations, support contacts — plus data-quality guardrails such as sample ratio mismatch.

Assumptions

Each guardrail has a direction ("higher is worse") and a tolerance δ fixed in advance.
Same randomisation and independence assumptions as the primary analysis.

How to interpret

Pass: the confidence bound rules out harm larger than δ.
Fail: the estimate itself shows harm beyond δ, or a clear significant deterioration.
Inconclusive: the bound includes harm beyond δ though the estimate is small — the test was not large enough to show safety.
With several guardrails, consider a multiple-comparison adjustment or treat them as alarms to investigate rather than automatic vetoes.

Common mistakes

Treating "no significant change" in a guardrail as proof of no harm (absence of evidence is not evidence of absence).
Choosing δ after seeing the data.
Letting a big win on the primary metric excuse an uninvestigated guardrail failure.

Example

Support-contact rate: A 300/10 000 (3.0%), B 330/10 000 (3.3%); tolerance δ = +0.5 points. The two-sided test gives p = 0.22 (not significant), yet the one-sided 95% upper bound for B − A is +0.71 points, above δ. Harm larger than the tolerance is not ruled out: the guardrail is inconclusive, not passed.