Track 13 · Resampling and Statistical Reasoning · Lesson 1 of 10

The bootstrap

Use bootstrap resampling to estimate a standard error and a confidence interval, and recognise the situations in which the bootstrap fails.

35 minAdvanced8 graded exercisesFree

A courier tests a new routing system and wants to know how precisely it has estimated the median delivery time — a statistic with no convenient standard-error formula. The bootstrap answers by resampling the data thousands of times. Used carelessly, it also produces confident-looking intervals for questions the data cannot answer.

A statistic computed from a sample would come out differently in another sample. Its standard error measures how much. For a mean there is a formula, $s/\sqrt{n}$ ; for a median, a trimmed mean, a ratio or a difference of medians there usually is not.

The bootstrap (Efron, 1979) estimates the sampling variability by computation. We cannot draw new samples from the population, so we draw new samples from the sample itself, treating it as a stand-in for the population.

The nonparametric bootstrap

Compute the statistic on the original sample of size $n$ : the estimate $\hat\theta$ .
Draw a resample: $n$ values taken from the sample with replacement. Some values appear twice or more, others not at all.
Compute the statistic on the resample: a replicate $\hat\theta^*$ .
Repeat steps 2–3 $B$ times (typically 2 000–10 000) and record the seed, so that anyone can reproduce the run.
Summarise the $B$ replicates: their standard deviation is the bootstrap SE; their middle 95% is the percentile interval.

\SE_{\text{boot}} = \sqrt{\frac{1}{B-1}\sum_{b=1}^{B}\big(\hat\theta^*_b - \bar\theta^*\big)^2}, \qquad \widehat{\text{bias}} = \bar\theta^* - \hat\theta

Bootstrap standard error and bias estimate

$\hat\theta$: the statistic on the original sample
$\hat\theta^*_b$: the statistic on the $b$ -th resample
$\bar\theta^*$: the mean of the $B$ replicates
$B$: the number of resamples (a simulation setting, not a sample size)

\text{percentile: } \big[\hat\theta^*_{(0.025)},\ \hat\theta^*_{(0.975)}\big] \qquad \text{basic: } \big[\,2\hat\theta - \hat\theta^*_{(0.975)},\ 2\hat\theta - \hat\theta^*_{(0.025)}\big]

Two 95% bootstrap intervals from the same replicates

$\hat\theta^*_{(q)}$: the $q$ quantile of the replicates
$2\hat\theta - \cdot$: the basic interval reflects the replicate quantiles around the estimate; it differs from the percentile interval when the bootstrap distribution is skewed or biased

Worked example

How precise is the median delivery time?

The delivery_independent dataset holds 76 deliveries routed by the new system. How precisely does their median estimate the median delivery time under the new system?

Estimate. The sample median is 37.95 minutes.
Resample. The engine drew $B = 10,000$ resamples of 76 deliveries with replacement (seed 2024) and took the median of each.
Standard error. The replicates have standard deviation 1.767: the bootstrap SE of the median is about 1.77 minutes. For comparison, the bootstrap SE of the mean is 1.735, close to the formula $s/\sqrt{n} = 1.727$ — a useful check that the method behaves.
Interval. The 2.5th and 97.5th percentiles of the replicate medians are 35.60 and 41.55: the 95% percentile interval.

Median 37.95 minutes, bootstrap SE 1.77, 95% percentile interval 35.60 to 41.55 minutes. A different seed would move these numbers slightly; a larger $B$ moves them less.

Bootstrap distribution of a statistic of the new-system delivery times.

Try this

Press Resample again a few times. The SE and the interval endpoints wobble in the second decimal: that is Monte Carlo error from using a finite $B$ , not uncertainty about the population.
Raise $B$ to 10 000. The wobble shrinks, but the interval does not get narrower: its width reflects $n = 76$ , which no amount of resampling can change.
Switch the statistic to the mean and compare the percentile interval with the t interval shown beside it. Then try the median: why does its bootstrap distribution look lumpy?

The same idea works for two independent groups: resample each group separately, within itself, and recompute the comparison. For the old system (80 deliveries) against the new (76), the difference in medians is 3.90 minutes, with bootstrap SE 2.73 and 95% percentile interval −1.60 to 9.20 minutes (B = 10,000, seed 2024).

Watch out: When the bootstrap fails

The bootstrap assumes that the sample looks like the population and that observations are independent. It breaks down when:

The sample is small. With $n$ below about 20, the sample is a poor picture of the population's tails and bootstrap intervals are typically too narrow.
The statistic depends on extremes. A resample can never exceed the sample maximum. Here, 62.6% of the resampled maxima of the new-system times equal the sample maximum (89.2 minutes) exactly — theory says about $1 - (1 - 1/n)^n \approx 63\%$ . The "distribution" is a spike, not a picture of uncertainty. Extreme quantiles (the 99.9th percentile of 300 values) suffer the same way.
Observations are dependent. Daily returns with volatility clustering, repeated measurements on the same person, or pupils in the same class: resampling single observations destroys the dependence and understates the SE. Resample whole blocks or clusters instead.
The sample is biased. Resampling a biased sample reproduces the bias with great precision.

Quick check

A bootstrap of a sample of 12 values with $B = 50\,000$ gives a narrow 95% interval. What is the main concern?

Exercises

The lesson is complete when every exercise is answered correctly. Hints are free; viewing a worked solution before solving means the exercise won't count towards mastery.

0 of 8 exercises correct (not saved — sign in to keep your progress)

Exercise 1 · Guided, difficulty 1 of 3

Bootstrap standard error from replicates

Not started

A sample of eight values is 29, 25, 20, 57, 16, 58, 55, 30 (mean 36.250).

Ten bootstrap resamples (with replacement) gave these means:

24.500, 29.750, 41.375, 35.125, 39.625, 26.125, 39.500, 37.750, 33.500, 29.250

Using these ten replicates, what is the bootstrap standard error of the mean? Use the $B - 1$ divisor and give 2 decimal places. (In practice you would use thousands of replicates; ten keep the arithmetic manageable.)

Exercise 2 · Practice, difficulty 2 of 3

Bias and the basic interval

Not started

Fifteen waiting times (minutes): 21, 22, 24, 25, 33, 34, 37, 38, 39, 43, 47, 49, 58, 59, 95 (sorted).

The sample median is $\hat\theta = 38.00$ . A bootstrap with $B = 2000$ resamples gave replicate medians with mean $\bar\theta^* = 38.39$ , 2.5th percentile 25.00 and 97.5th percentile 49.00.

What is the bootstrap estimate of the bias of the median?
Compute the 95% basic bootstrap interval (lower and upper limit).

Give all answers to 2 decimal places.

Exercise 3 · Application, difficulty 2 of 3

Old versus new routing: medians

Not started

Use delivery_independent. Compute the median delivery time for the old system and for the new system, and the difference old − new. Give each to 2 decimal places.

(A two-sample bootstrap of this difference — each group resampled within itself, $B = 10,000$ , seed 2024 — gave SE 2.73 minutes and a 95% percentile interval from −1.60 to 9.20. You will interpret it in the next exercise.)

Data: delivery_independent · Open in the lab to explore and calculate.

Exercise 4 · Interpretation, difficulty 2 of 3

Reading a bootstrap interval

Not started

For the routing comparison, the difference in median delivery time (old − new) is 3.90 minutes. A two-sample bootstrap ( $B = 10,000$ , seed 2024) gives a 95% percentile interval from −1.60 to 9.20 minutes. Which statement is the best reading?

Exercise 5 · Challenge, difficulty 3 of 3

Why the bootstrap fails for the maximum

Not started

A sample has $n = 55$ distinct values. A bootstrap resample draws $n$ values with replacement.

What is the probability that a resample contains the sample maximum at least once — so that the resampled maximum equals the sample maximum exactly? (4 decimal places)
With $B = 1000$ resamples, about how many replicate maxima will equal the sample maximum? (nearest whole number)

Exercise 6 · Practice, difficulty 2 of 3

Can you trust the bootstrap here?

Not started

For each analysis, decide whether a standard (resample-individual-observations) bootstrap is a reasonable way to get a standard error or interval.

Exercise 7 · Practice, difficulty 1 of 3

Steps of a two-sample bootstrap

Not started

Put the steps of a two-sample percentile bootstrap for a difference in medians in the correct order.

Exercise 8 · Interpretation, difficulty 2 of 3

Report the bootstrap honestly

Not started

Write two or three sentences for the courier's operations manager reporting the bootstrap result for the new system alone: median delivery time 37.95 minutes, bootstrap SE 1.77, 95% percentile interval 35.60 to 41.55 minutes ( $n = 76$ deliveries, $B = 10,000$ , seed 2024). Say what the interval means and state one assumption the bootstrap relies on.

What you can now do

Compute a bootstrap standard error, bias estimate, percentile interval and basic interval
Bootstrap a difference between two independent groups by resampling within each group
Separate Monte Carlo error (shrinks with B) from sampling uncertainty (shrinks only with n)
Recognise when the bootstrap fails: small samples, extremes, dependent or biased data