Multiple-comparison adjustments (Bonferroni, Holm, Hochberg, Benjamini–Hochberg)

Adjust p-values when running several tests so that either the family-wise error rate (FWER) or the false discovery rate (FDR) stays at α.

\tilde p_i^{\text{Bonf}} = \min(1,\ m\,p_i)

Bonferroni (FWER).

$m$: Number of tests in the family.
$p_i$: Unadjusted p-value of test i.

\tilde p_{(i)}^{\text{Holm}} = \max_{j \le i}\ \min\{1,\ (m - j + 1)\,p_{(j)}\}

Holm step-down (FWER; uniformly more powerful than Bonferroni, the engine's default).

$p_{(1)} \le \dots \le p_{(m)}$: The p-values sorted in increasing order.

\tilde p_{(i)}^{\text{Hoch}} = \min_{j \ge i}\ \min\{1,\ (m - j + 1)\,p_{(j)}\},\qquad \tilde p_{(i)}^{\text{BH}} = \min_{j \ge i}\ \min\left\{1,\ \frac{m}{j}\,p_{(j)}\right\}

Hochberg step-up (FWER, assumes independence or positive dependence) and Benjamini–Hochberg (FDR).

$j$: Rank of a p-value in the sorted list.

When to use

Whenever several hypotheses are tested and any one "hit" would be reported: many variants against a control, many metrics or segments, all pairs of groups. Decide the family and the method before looking at results.

FWER (Bonferroni, Holm, Hochberg): probability of at least one false positive ≤ α. Use for confirmatory claims and small families.
FDR (Benjamini–Hochberg): expected share of false positives among the rejections ≤ α. Use for screening many hypotheses.

Assumptions

Bonferroni and Holm are valid under any dependence. Hochberg and BH assume independent or positively dependent tests (Benjamini–Yekutieli covers arbitrary dependence at a cost in power).

How to interpret

Compare adjusted p-values with α as usual. With 20 independent tests at α = 0.05 and all nulls true, the chance of at least one false positive is $1 - 0.95^{20} \approx 0.64$ — that is what adjustment prevents.

Common mistakes

Adjusting only the tests that "look interesting" (the family must include every test run).
Using Bonferroni when Holm is available (Holm rejects everything Bonferroni does, and sometimes more).
Reading an FDR-adjusted significant result as having FWER protection.

Example

Four variant-vs-control tests: p = 0.01, 0.04, 0.03, 0.20.

Bonferroni: 0.04, 0.16, 0.12, 0.80 → one rejection at 0.05.
Holm: 0.04, 0.09, 0.09, 0.20 → one rejection.
Hochberg: 0.04, 0.08, 0.08, 0.20 → one rejection.
BH: 0.04, 0.053, 0.053, 0.20 → one rejection at FDR 5% (three at 10%).