Track 8 · Hypothesis Testing · Lesson 1 of 14

The logic of a hypothesis test

Explain how a hypothesis test weighs data as evidence against a sceptical claim, and what its two possible conclusions do and do not mean.

25 minIntermediate7 graded exercisesFree

A colleague says she can taste whether a coffee is decaf. In a blind test she labels 14 of 16 cups correctly. Is that skill, or could a lucky guesser do the same? A hypothesis test turns this everyday argument into a precise, repeatable procedure — and it is the same argument behind drug approvals, A/B tests and quality audits.

Data always vary. Two groups of customers never have exactly the same average spend, and a fair coin rarely lands heads exactly half the time. So whenever we see a pattern, we must ask: could ordinary chance variation alone have produced something like this?

A hypothesis test answers that question with an argument by contradiction:

Start from a sceptical claim — nothing interesting is going on. This is the null hypothesis, H0.
Work out what data we would expect to see if H0 were true.
Measure how surprising the observed data would be under H0.
If the data would be very surprising under H0, we have evidence against H0. If not, the data are compatible with H0 and we have no convincing evidence against it.

Worked example

The decaf taster

In a blind test with 16 cups, each cup is decaf or regular at random (a fair coin decides). The taster labels 14 of the 16 cups correctly. Is this convincing evidence that she does better than guessing?

Sceptical claim. H0: she is guessing, so each cup has probability $p = 0.5$ of being labelled correctly. The claim we hope to find evidence for is H1: $p > 0.5$ .
What H0 predicts. If she is guessing, the number of correct labels $X$ follows a Binomial(16, 0.5) distribution. On average she would get 8 right, and 9 or 10 would be unremarkable.
How surprising is the result? We ask how often a guesser would do at least as well as 14 correct: $P(X \ge 14) = P(14) + P(15) + P(16) = 0.0021$ . A guesser does this well only about 2 times in 1 000.
Weigh the evidence. Either something very unusual happened by luck, or H0 is wrong. A probability this small is strong evidence against H0.

The result would be very rare for a guesser (probability about 0.002), so the test gives strong evidence that her success rate is above 50%. It does not tell us how much better than guessing she is — that needs an estimate with a confidence interval.

The probability we just computed — the chance, assuming H0 is true, of a result at least as extreme as the one observed — is called the p-value. Small p-values mean the data sit uncomfortably with H0. The next lessons make this precise; for now notice the direction of the reasoning: we never compute the probability that H0 is true. We assume H0 and ask how well the data fit.

Courtroom	Hypothesis test
Presumption of innocence	Null hypothesis H0, assumed true unless the data say otherwise
Evidence presented	The data, summarised by a test statistic
Beyond reasonable doubt	Significance level α, the standard of evidence fixed in advance
Guilty verdict	Reject H0
Not guilty verdict	Do not reject H0 (not “H0 is true”)
Convicting an innocent person	Type I error: rejecting a true H0
Acquitting a guilty person	Type II error: not rejecting a false H0

The analogy, term by term

Suppose instead she had labelled 9 of 16 correctly. A guesser gets 9 or more correct with probability $P(X \ge 9) = 0.4018$ — about 40% of the time. Nine correct is entirely ordinary for a guesser, so the data give no convincing evidence of skill. That is not the same as evidence that she has no skill: with only 16 cups, even a real but modest ability (say 65% correct) would often produce scores like 9 or 10.

The workflow used in every method lesson of this track

Question. State the research question in context, and the parameter it is about.
Hypotheses and standard of evidence. Write H0 and H1, choose one- or two-sided, and fix α — all before looking at the data.
Design and assumptions. Check how the data were collected (random sampling or assignment, independence) and the conditions the method needs.
Compute. Find the test statistic, its null distribution and the p-value.
Estimate. Report the effect size and a confidence interval: how big is the effect, and how precisely is it known?
Conclude in context. Reject or do not reject H0, then say what that means for the original question, without overclaiming.

Quick check

A test of a new teaching method ends with “do not reject H0” (H0: the method makes no difference to the mean score). Which conclusion is correct?

Exercises

The lesson is complete when every exercise is answered correctly. Hints are free; viewing a worked solution before solving means the exercise won't count towards mastery.

0 of 7 exercises correct (not saved — sign in to keep your progress)

Exercise 1 · Guided, difficulty 1 of 3

How surprising is a guesser's score?

Not started

A wine taster claims she can tell which of two glasses is the more expensive wine. In 13 blind trials (the expensive glass placed left or right by a coin flip) she is right 11 times.

If she were simply guessing, the number of correct answers $X$ would follow a Binomial(13, 0.5) distribution. Compute $P(X \ge 11)$ , the probability that a guesser does at least this well. Give 4 decimal places.

Exercise 2 · Practice, difficulty 1 of 3

Put the reasoning in order

Not started

A quality engineer suspects a filling machine now puts more than the labelled 500 ml into bottles. Put the steps of her hypothesis test in the correct order.

Exercise 3 · Practice, difficulty 1 of 3

Courtroom and test

Not started

Match each courtroom idea with its counterpart in a hypothesis test.

Exercise 4 · Interpretation, difficulty 2 of 3

What does “not guilty” mean here?

Not started

The decaf taster is tested again with 16 new cups and gets 9 correct. If she were guessing, $P(X \ge 9) = 0.4018$ . Which conclusion is correct?

Exercise 5 · Interpretation, difficulty 2 of 3

Sound conclusion or overclaim?

Not started

A clinic tested whether a reminder text message changes the no-show rate for appointments. Sort each statement into the right category.

p = 0.003: the data give strong evidence that the reminder changes the no-show rate.

p = 0.003: the test proves that reminders work.

p = 0.41: the reminder has no effect on no-shows.

p = 0.41: the data are compatible with no effect; the study does not show whether the reminder helps.

p = 0.003: there is a 0.3% probability that the null hypothesis is true.

p = 0.003: if reminders made no difference, a difference at least this large would occur only about 3 times in 1 000 studies like this.

Exercise 6 · Practice, difficulty 2 of 3

The sceptic's threshold

Not started

A taster will be given 14 cups. Before the test, the sceptics agree they will only doubt the guessing explanation if a guesser would reach her score (or better) with probability at most 0.05.

What is the smallest number of correct cups that meets this standard? Also give $P(X \ge \text{that number})$ for a guesser, to 4 decimal places.

Exercise 7 · Interpretation, difficulty 2 of 3

Explain a non-significant result

Not started

In the second test the decaf taster labelled 9 of 16 cups correctly. A guesser does at least this well with probability 0.40.

In two or three sentences, write the conclusion for the café owner who ran the test. Say what the probability tells us, what we conclude about her claim, and what the result does not show.

What you can now do

Describe a hypothesis test as an argument by contradiction from a sceptical null hypothesis
Compute how surprising a result would be under H0 in a simple binomial setting
Explain why “do not reject H0” is not the same as “H0 is true”
Use the courtroom analogy and state where it breaks down