Track 1 · Statistical Thinking · Lesson 10 of 10
Critiquing statistical claims
Evaluate a statistical claim systematically — source, sample, design, measure, comparison, effect size and uncertainty — and rewrite it honestly.
“Employees who use our wellness app take 40% fewer sick days!” says a vendor's brochure sent to your HR department. The number may be accurate and the claim still misleading. A short, systematic checklist lets you see what the evidence really supports before the company signs a contract.
Every lesson in this track gives you one question to ask of a statistic. Put together, they form a checklist you can apply to any claim — in a news article, a vendor pitch, a dashboard or your own report.
Seven questions for any statistical claim
- Source — Who produced the number, and do they benefit from a particular conclusion? Is the method published?
- Sample — Who or what was measured, how were they selected, and how many? Do they represent the population the claim is about? (Bias)
- Design — Observational or randomised? Could a confounder explain the result? Does the claim say cause when the design shows only association? (Confounding)
- Measure — How was the outcome defined and measured? Is it a valid measure of what the claim talks about? (Where data come from)
- Comparison — Compared with what? A claim with no comparison group, or with a hand-picked one, says little.
- Size — How big is the effect in absolute terms, not only relative ones? Is it big enough to matter?
- Uncertainty — Is there a margin of error or interval? Could chance variation explain the result? (Uncertainty and significance)
Worked example
The wellness-app brochure
The vendor's footnote says: “Among 300 employees at a client company, the 90 who chose to use the app averaged 3 sick days last year; the 210 non-users averaged 5.” Apply the checklist.
Source: the vendor, who profits from the conclusion, and one client company that may be unusual.
Sample and design: users chose the app. People who sign up for wellness apps are often already healthier or more health-conscious — a likely confounder. This is an observational comparison.
Size: “40% fewer” is relative: . In absolute terms it is 2 days per employee per year — useful to know when weighing the price.
Uncertainty: no information on how variable sick days are; with 90 users, a few long illnesses among non-users could drive the difference.
Honest version: “At one client, employees who chose to use the app took on average 2 fewer sick days than those who did not. Because users selected themselves, this does not show the app reduced sick leave.”
Relative versus absolute change is the most common way a true number misleads. If a risk falls from to :
- the risk (or rate) in the comparison group, or before
- the risk (or rate) in the treated group, or after
A screening programme that lowers the risk of a condition from 2 in 1,000 to 1 in 1,000 “halves the risk” (relative change −50%) — but the absolute reduction is 0.1 percentage points: one person in 1,000. Both numbers are correct; reporting only the relative one exaggerates the benefit. Report both.
The same logic applies to counts versus rates. A large city has more burglaries than a small one simply because it has more homes. Compare rates — per 1,000 homes, per 100,000 residents, per million passenger-kilometres — before ranking places, products or teams.
| Red flag | What to ask |
|---|---|
| Only relative changes (“doubles”, “40% fewer”) | What are the absolute numbers? |
| Counts compared across groups of different sizes | What are the rates? |
| A time window that starts at an unusual low or high | What does a longer series show? |
| Only successes are studied (surviving firms, winning funds) | What happened to the ones that failed? |
| “Linked to”, then “causes” | Was it randomised? What else differs between the groups? |
| A precise-looking number with no margin of error | How much would it change with a new sample? |
| A small or self-selected sample | Who is missing, and why? |
Quick check
A fund manager advertises that the five funds she launched ten years ago that are still open have all beaten the market. Which red flag applies most?
Exercises
The lesson is complete when every exercise is answered correctly. Hints are free; viewing a worked solution before solving means the exercise won't count towards mastery.
0 of 7 exercises correct (not saved — sign in to keep your progress)
Exercise 1 · Practice, difficulty 1 of 3
Spot the flaw
Match each claim to its main weakness.
Exercise 2 · Guided, difficulty 2 of 3
“Halves the risk”
A (fictional) health campaign reports that a new vaccination programme reduced the risk of hospitalisation from 18 per 1,000 people to 7 per 1,000.
(a) What is the absolute reduction in risk, in percentage points (2 decimal places)? (b) What is the relative reduction, in percent (1 decimal place)?
Exercise 3 · Practice, difficulty 2 of 3
Counts or rates?
A newspaper headline reads: “City North is the burglary capital: 6,153 burglaries last year, against only 3,699 in City South.” City North has 2,100,000 residents and City South 900,000.
Compute each city's burglary rate per 100,000 residents (1 decimal place).
Exercise 4 · Practice, difficulty 2 of 3
Questions worth asking
A food-delivery platform announces: “Restaurants that joined our premium listing saw orders rise by 35%.” Select all the questions that would help you judge the claim.
Exercise 5 · Practice, difficulty 2 of 3
Classify the weakness
Classify the main weakness of each claim.
Exercise 6 · Interpretation, difficulty 2 of 3
Which evidence is strongest?
A school district wants to know whether a reading app improves Grade-4 reading scores. Which piece of evidence would most strongly support a causal claim?
Exercise 7 · Interpretation, difficulty 3 of 3
Critique a press release
A (fictional) company's press release says: “Our energy drink boosts productivity by 25%! In our study, the 40 office workers who chose to drink it every morning completed 25% more tasks than colleagues who did not.” Using the checklist, write three to five sentences critiquing the claim and propose a more honest statement.
What you can now do
- Apply a seven-question checklist — source, sample, design, measure, comparison, size, uncertainty — to any claim
- Convert between relative and absolute changes and report both
- Compare rates rather than counts when groups differ in size
- Rewrite an overreaching claim into one the evidence supports