Track 1 · Statistical Thinking · Lesson 9 of 10

Uncertainty and significance

Explain sampling variability, compute and interpret an approximate margin of error for a proportion, and distinguish statistical significance from practical importance.

35 minFoundation7 graded exercisesFree

Two polls of the same city a week apart put support for a new metro line at 54% and 49%. Did opinion shift, or is that just what samples do? Before reading anything into a number from a sample, you need a sense of how much it would move if the sample were drawn again.

Different random samples from the same population give different statistics. This sampling variability is not a mistake; it is a predictable consequence of looking at part of the population. Two facts about it drive everything that follows:

Estimates from random samples scatter around the true value (no bias), and
the scatter shrinks as the sample size grows — but slowly, in proportion to $1/\sqrt{n}$ .

A margin of error turns that idea into a number. For a sample proportion from a simple random sample, an approximate 95% margin of error is

\text{MoE} \approx 1.96\sqrt{\frac{\hat{p}\,(1-\hat{p})}{n}}

Approximate 95% margin of error for a proportion (reasonable when $n\hat{p}$ and $n(1-\hat{p})$ are both at least about 10)

$\hat{p}$: the sample proportion
$n$: the sample size
$\sqrt{\hat{p}(1-\hat{p})/n}$: the standard error: the typical size of the sampling error of $\hat{p}$
$1.96$: the multiplier for 95% confidence (from the normal distribution)

Worked example

A city poll

A random sample of 1,000 residents finds 540 support the metro line. Compute the margin of error and the resulting interval.

$\hat{p} = 540/1{,}000 = 0.54$ .
Standard error $= \sqrt{0.54 \times 0.46 / 1{,}000} = \sqrt{0.0002484} = 0.01576$ .
MoE $= 1.96 \times 0.01576 = 0.0309$ , about 3.1 percentage points.
Interval: $54\% \pm 3.1$ , i.e. from 50.9% to 57.1%. Every value in that range is reasonably consistent with the data.

54% ± 3.1 percentage points.

Worked example

Did opinion shift between the two polls?

A week later a second, independent random sample of 1,000 residents gives 49% support (margin of error about ±3.1 points, interval 45.9% to 52.1%). The two intervals overlap. Does that mean the 5-point drop could just be chance?

Each poll has its own sampling error, and the difference inherits both. The standard error of a difference between two independent estimates is $\sqrt{\text{SE}_1^2 + \text{SE}_2^2} = \sqrt{0.54 \times 0.46/1{,}000 + 0.49 \times 0.51/1{,}000} = 0.0223$ .
Margin of error for the change: $1.96 \times 0.0223 = 0.044$ , about ±4.4 points — larger than either poll's own margin (3.1), but smaller than the 6.2 points you would get by simply adding the two margins.
Compare: the observed drop of 5 points is just outside ±4.4, so it is (narrowly) larger than chance variation alone would usually produce. Overlapping intervals did not settle the question: checking for overlap is a too-strict test of a difference.
And still be modest: a drop this close to the margin is weak evidence on its own; a third poll, or a change in question wording between polls, would matter a lot.

The change has a margin of error of about ±4.4 points, so a 5-point drop is narrowly beyond chance — despite the overlapping intervals.

Statistical significance. When we compare groups — two versions of a web page, a treatment and a placebo — some difference will appear by chance alone. A difference is called statistically significant when it is larger than chance variation would plausibly produce if there were no real effect. Track 8 (Hypothesis Testing) makes this precise with p-values. For now, two warnings:

Significant is not the same as important. With a huge sample, a tiny, commercially irrelevant difference can be statistically significant. Always look at the size of the effect and ask whether it matters in context — cost, risk, benefit.
Not significant is not the same as no effect. With a small sample, even a large real effect may not stand out from the noise. Absence of evidence is not evidence of absence.

Quick check

A pricing test on 20 million visitors (10 million per page) finds that a new page lifts conversion from 3.10% to 3.12%, and the difference is statistically significant. What follows?

Exercises

The lesson is complete when every exercise is answered correctly. Hints are free; viewing a worked solution before solving means the exercise won't count towards mastery.

0 of 7 exercises correct (not saved — sign in to keep your progress)

Exercise 1 · Guided, difficulty 2 of 3

Margin of error for a poll

Not started

A random sample of 500 registered patients at a health network found that 345 would use video consultations. Compute the approximate 95% margin of error for the proportion, $1.96\sqrt{\hat{p}(1-\hat{p})/n}$ , in percentage points (1 decimal place).

Exercise 2 · Practice, difficulty 2 of 3

Bigger sample, smaller margin

Not started

A customer survey of 300 people has a margin of error of ±4.8 percentage points. The marketing director wants a more precise estimate and increases the sample to 2,700 people (9 times as many), with the same method and a similar proportion. What will the margin of error be, approximately? Give 2 decimal places.

Exercise 3 · Guided, difficulty 2 of 3

How many to survey?

Not started

A regulator wants to estimate the share of small businesses that use electronic invoicing with a 95% margin of error of at most ±3.0 percentage points. With no prior idea of the share, it plans for the worst case $\hat{p} = 0.5$ . Solve $1.96\sqrt{0.5 \times 0.5 / n} \le 0.030$ for $n$ , and give the smallest whole number of businesses needed.

Exercise 4 · Interpretation, difficulty 2 of 3

Reading a poll

Not started

A random-sample poll finds 52% of voters support a referendum proposal, with a 95% margin of error of ±3.1 percentage points. Which statement is correct?

Exercise 5 · Interpretation, difficulty 2 of 3

Significant but small

Not started

An online retailer tests a redesigned product page on 10 million visitors (randomly split, 5 million per version). Conversion is 4.03% with the new page and 4.00% with the old one, and the difference is statistically significant. Redesigning all product pages would cost 3 million SAR. What should the analyst say?

Exercise 6 · Practice, difficulty 3 of 3

Sound or unsound interpretation?

Not started

Classify each statement as a sound interpretation or a misinterpretation.

“The difference was not statistically significant, so the two training methods are equally effective.”

“If we repeated this survey many times, about 95% of the intervals built this way would contain the true proportion.”

“The result is highly significant, so the effect must be large.”

“A larger random sample would give a narrower margin of error.”

“Our sample of 50,000 website visitors has a margin of error of ±0.4 points, so the estimate is free of bias.”

“The new process cut average handling time by 6 seconds; the difference is statistically significant but too small to justify retraining costs.”

Exercise 7 · Interpretation, difficulty 3 of 3

A permit-processing pilot

Not started

A ministry piloted a new online permit system across many offices. Average processing time fell from 12.0 days to 11.8 days, and with tens of thousands of applications the difference is statistically significant. A press officer drafts: “Significant breakthrough: the new system dramatically speeds up permits.” In two to four sentences, explain what the result does and does not show, and how you would rephrase it.

What you can now do

Explain sampling variability and why it shrinks with the square root of the sample size
Compute and interpret an approximate 95% margin of error for a proportion
Plan a sample size for a target margin of error
Distinguish statistical significance from practical importance, and non-significance from “no effect”