Track 1 · Statistical Thinking · Lesson 5 of 10
Observational studies and experiments
Distinguish observational studies from randomised experiments, identify explanatory and response variables, and explain why random assignment supports causal conclusions.
Employees who attend a voluntary stress-management course report fewer sick days the following year. Did the course cause that — or were the people who signed up already healthier and more motivated? Whether you can answer depends entirely on how the study was designed.
Most studies ask whether one variable affects another. The explanatory variable (or treatment, factor) is the one we think does the influencing; the response variable (or outcome) is the one we think is influenced. In “does the course reduce sick days?”, course attendance is explanatory and sick days are the response.
There are two fundamentally different ways to collect data on such a question.
| Observational study | Experiment | |
|---|---|---|
| Who decides the treatment? | The subjects themselves, or circumstances | The researcher |
| Example | Compare sick days of employees who chose to attend the course with those who did not | Randomly assign half of the volunteers to the course now and half to a later course |
| Can it show causation? | Only with strong extra assumptions | Yes, if well designed and randomised |
| Main threat | Confounding: groups differ in other ways | Poor control, lack of blinding, dropout |
In an observational study the groups form themselves. People who choose a course, a diet or a product usually differ in many other ways — age, motivation, income, health — and any of those differences could explain the outcome.
In an experiment the researcher imposes the treatment. The key ingredient is random assignment: using chance (a random number generator, not judgement) to decide who gets which treatment. Randomisation makes the groups alike, on average, in every characteristic — including ones nobody measured. If the groups then differ in outcome by more than chance would explain, the treatment is the plausible cause.
Ingredients of a well-designed experiment
- Comparison: a control group that receives no treatment, a placebo or the current standard — so you can see what would have happened anyway.
- Random assignment of subjects to groups, so the groups are comparable.
- Blinding where possible: participants (single-blind) and also the people measuring outcomes (double-blind) do not know who got what, so expectations cannot shape the results. A placebo — a dummy treatment that looks identical — makes blinding possible.
- Replication: enough subjects that chance differences between groups are small.
Worked example
A blood-pressure trial (fictional)
The bp_trial dataset is a simulated double-blind trial: 200 adults with raised systolic blood pressure were randomly assigned, 100 to a new tablet and 100 to a placebo. Did randomisation produce comparable groups, and what happened to blood pressure?
Comparable at the start. Mean baseline systolic pressure was 153.6 mmHg in the treatment arm and 153.9 mmHg in the placebo arm; the arms also had similar ages (56.7 vs 57.5 years) and numbers of women (47 vs 46). Randomisation did its job.
Change after 8 weeks (among the 91 and 90 participants who completed): the treatment arm fell by 10.98 mmHg on average, the placebo arm by 1.49 mmHg.
The comparison. The placebo arm also improved a little — through the placebo effect and regression to the mean (people were enrolled because a single reading was high, and such readings tend to come down). Only the difference between arms, about mmHg, estimates the effect of the tablet.
A caution about dropouts. Nine treatment and ten placebo participants did not finish. Randomisation balanced the arms as allocated; if people left one arm for reasons linked to their blood pressure (side effects, say), the completers might no longer be comparable. Here dropout was similar in both arms, but a full trial report would also analyse everyone as randomised (called intention to treat), so that the comparison keeps the balance randomisation created.
Is 9.5 mmHg a lot? Whether an effect of that size matters for patients is judged by clinicians against guidelines for the condition, not by the statistics alone.
The groups were balanced by randomisation; among completers, the tablet's estimated effect is about 9.5 mmHg more reduction than placebo.
Show the data as a table
| Group | n | Lower whisker | Q1 | Median | Q3 | Upper whisker | Outliers | Mean |
|---|---|---|---|---|---|---|---|---|
| Treatment | 100 | 140 | 146.75 | 153 | 159 | 172 | 2 | 153.57 |
| Placebo | 100 | 140 | 145.75 | 153 | 160 | 180 | 2 | 153.9 |
Quick check
Researchers compare the test scores of pupils at schools that chose to adopt a new reading programme with pupils at schools that did not. What kind of study is this?
Exercises
The lesson is complete when every exercise is answered correctly. Hints are free; viewing a worked solution before solving means the exercise won't count towards mastery.
0 of 8 exercises correct (not saved — sign in to keep your progress)
Exercise 1 · Practice, difficulty 1 of 3
Observational or experiment?
Classify each study.
Exercise 2 · Practice, difficulty 1 of 3
The language of experiments
Match each term to its description.
Exercise 3 · Application, difficulty 2 of 3
Change in each arm
Using the bp_trial data, keep only participants who completed the trial (week8_sbp is not missing). For each one compute change = week8_sbp − baseline_sbp. What is the mean change in each arm? Give both to 2 decimal places (a fall is negative).
Data: bp_trial · Open in the lab to explore and calculate.
Exercise 4 · Guided, difficulty 1 of 3
An onboarding experiment
A banking app randomly assigned new users to the current onboarding flow (A) or a simplified flow (B). In group A, 600 of 1,500 users completed account set-up; in group B, 761 of 1,620 did.
By how many percentage points is B's completion rate higher than A's? Give 1 decimal place.
Exercise 5 · Interpretation, difficulty 2 of 3
Gym members and sick days
An HR department finds that employees who use the company gym took on average 3 fewer sick days last year than those who do not. The HR director wants to announce that “using the gym reduces sick leave by 3 days a year”. What is the best assessment?
Exercise 6 · Practice, difficulty 2 of 3
Designing an experiment
A logistics firm wants to know whether a new route-planning app reduces drivers' delivery times. Put the steps of a sound experiment in order.
Exercise 7 · Interpretation, difficulty 3 of 3
Generalise or explain?
A university randomly selects 1,000 of its 30,000 students and asks how many hours they study per week and what their grade average is. Students who study more have higher grades. Which conclusion is justified?
Exercise 8 · Interpretation, difficulty 3 of 3
Why did the placebo arm improve?
In the blood-pressure trial, systolic pressure fell by about 11 mmHg in the treatment arm — but it also fell by about 1.5 mmHg in the placebo arm. In two to four sentences, explain why the placebo arm improved and why the trial needed a randomised placebo group to estimate the tablet's effect.
What you can now do
- Identify explanatory and response variables in a study
- Tell an observational study from a randomised experiment
- Explain how random assignment, control groups and blinding support causal conclusions
- Distinguish random sampling (generalisation) from random assignment (causation)