Track 1 · Statistical Thinking · Lesson 7 of 10
Bias and representative samples
Recognise selection, nonresponse, response and measurement bias, and explain why a larger biased sample does not fix bias.
A news website asks readers whether fuel subsidies should be cut, and 48,000 people vote. A polling firm asks 1,000 randomly chosen adults the same question. The two results disagree by 20 points. Which do you trust — and why does the huge number of votes not help?
A sampling method is biased if it tends to miss the population value in the same direction, sample after sample. Bias is different from sampling variability: variability makes estimates scatter around the truth; bias moves the whole scatter away from it. A representative sample is one whose make-up mirrors the population in the ways that matter for the question.
| Bias | What goes wrong | Example |
|---|---|---|
| Selection bias / undercoverage | Some parts of the population are less likely (or unable) to be sampled | A phone survey drawn from landline numbers misses households with only mobiles |
| Convenience sampling | Sampling whoever is easiest to reach | Interviewing shoppers at one mall at 11 a.m. on a weekday |
| Voluntary response | People choose themselves, and those with strong views respond most | An online poll on a news site |
| Nonresponse bias | Selected people who do not respond differ from those who do | Less engaged employees skip the engagement survey |
| Response bias | People answer inaccurately — because of wording, social desirability or memory | “Don't you agree that our excellent service…?”; under-reported smoking |
| Measurement bias | The instrument or procedure is systematically off | A blood-pressure cuff that is too small for the arm reads high |
Here is nonresponse in a (synthetic) company. All 1,400 Palmstone Group employees were invited to an engagement survey. Response rates ranged from 45% in Operations to 80% in Human Resources — and the departments that responded least were also the least engaged.
Dataset
Engagement survey population and response rates
6 rows · one row = one department · synthetic data
| Department | Headcount (employees) | Responses (responses) | Response rate (%) |
|---|---|---|---|
| Operations | 420 | 189 | 45 |
| Customer Service | 330 | 165 | 50 |
| Sales | 260 | 143 | 55 |
| Technology | 210 | 151 | 71.9 |
| Finance | 120 | 82 | 68.3 |
| Human Resources | 60 | 48 | 80 |
Headcount and responses for each department.
Worked example
Who is over-represented?
Compare Operations' share of the workforce with its share of the survey responses, and see what that does to the overall result.
Share of staff: Operations has 420 of 1,400 employees, .
Share of responses: it supplied 189 of 778 responses, . Human Resources, by contrast, is 4.3% of staff () but 6.2% of responses ().
Effect on the estimate: the mean answer to “I am proud to work for this organisation” is 3.18 in Operations but 3.81 in HR. Because the more positive departments are over-represented, the unweighted mean of all responses (3.40) overstates engagement. Weighting each department's mean by its headcount gives 3.35.
Operations is under-represented (24.3% of responses vs 30.0% of staff), so the raw survey mean is biased upwards.
Weighting (as in the example) can correct for nonresponse that is linked to known characteristics such as department. It cannot correct for differences you did not measure: if, within each department, unhappy employees were also less likely to respond, the weighted estimate is still too high. Prevention — a high response rate — beats cure.
Quick check
A gym surveys members who attend classes at least three times a week about satisfaction with the timetable. What is the main problem?
Exercises
The lesson is complete when every exercise is answered correctly. Hints are free; viewing a worked solution before solving means the exercise won't count towards mastery.
0 of 7 exercises correct (not saved — sign in to keep your progress)
Exercise 1 · Practice, difficulty 1 of 3
Name the bias
Classify the main source of bias in each situation.
Exercise 2 · Application, difficulty 1 of 3
Who answered?
Using engagement_population, compare Customer Service's share of the whole workforce (headcount) with its share of all survey responses. Give both as percentages to 1 decimal place.
Data: engagement_population · Open in the lab to explore and calculate.
Exercise 3 · Challenge, difficulty 3 of 3
Correcting for nonresponse
The engagement survey over-represents the more engaged departments. Estimate the mean answer to q6_intend_stay (“I intend to be working here in 12 months' time”, 1–5) for all employees by weighting:
- compute the mean of
q6_intend_stayin each department (ignore skipped answers); - weight each department mean by the department's headcount:
| Department | Headcount |
|---|---|
| Operations | 420 |
| Customer Service | 330 |
| Sales | 260 |
| Technology | 210 |
| Finance | 120 |
| Human Resources | 60 |
Give the weighted mean to 2 decimal places. (Treating the 1–5 codes as numbers is a simplification commonly used for tracking.)
Data: engagement_survey · Open in the lab to explore and calculate.
Exercise 4 · Interpretation, difficulty 2 of 3
48,000 votes or 1,000 random adults?
A news website's online poll (48,000 votes) finds 71% oppose cutting fuel subsidies. A polling firm's random sample of 1,000 adults finds 52% oppose. Which estimate is more trustworthy for the adult population, and why?
Exercise 5 · Practice, difficulty 2 of 3
How nonresponse distorts a result
A utility company emails all 40,000 of its business customers about a new tariff. In fact 55% of them prefer the new tariff. Customers who prefer it reply at a rate of 35%; those who do not reply at 20%.
What percentage of the replies will favour the new tariff? Give 1 decimal place.
Exercise 6 · Practice, difficulty 1 of 3
Fix the question
A municipality wants to measure support for a new bus lane. Which wording is least likely to cause response bias?
Exercise 7 · Interpretation, difficulty 3 of 3
Explain the engagement bias
The HR director of the (fictional) Palmstone Group wants to report the unweighted survey average as “employee engagement across the company”. Response rates were 45% in Operations and 80% in HR, and the low-response departments are less engaged. In two to four sentences, explain why the unweighted average is misleading, in which direction, and what can and cannot be done about it.
What you can now do
- Name the bias in a sampling or measurement procedure and its likely direction
- Explain why a larger biased sample gives a precise answer to the wrong question
- Compare a sample's make-up with the population's to detect nonresponse bias
- Correct for known imbalances by weighting, and state what weighting cannot fix