Hypothesis testing: the framework

A hypothesis test measures how surprising the data would be if a stated null hypothesis about a parameter were true, and turns that into a decision at a pre-chosen significance level.

\text{test statistic} = \frac{\text{estimate} - \text{null value}}{\text{standard error of the estimate}}

Most tests in this reference share this shape: a standardised distance from the null value.

$\text{estimate}$: The sample statistic, e.g. $\xbar$ , $\phat$ or $\xbar_1 - \xbar_2$ .
$\text{null value}$: The parameter value stated by $H_0$ , e.g. $\mu_0$ or $0$ .
$\text{standard error}$: The estimated standard deviation of the estimate across repeated samples.

\text{reject } H_0 \iff p \le \alpha

The decision rule.

$p$: The p-value of the observed statistic.
$\alpha$: The significance level, fixed before seeing the data (often 0.05).

When to use

You have a specific claim about a population parameter (a mean, a proportion, a difference) and want to know whether the data provide evidence against it.
Pair every test with an estimate and a confidence interval: the test answers "is there evidence of a difference?", the interval answers "how big might it be?".

Assumptions

Every test has a model: how the data were collected (random sampling or random assignment, independence) and, for some tests, a distributional shape. The p-value is only as trustworthy as that model.

How to interpret

The workflow:

State the hypotheses about a parameter (never a statistic). $H_0$ is the "no effect / no difference" claim with an equality, e.g. $H_0: \mu = 30$ ; $H_1$ is two-sided ( $\mu \ne 30$ ) or one-sided ( $\mu > 30$ ), chosen before looking at the data.
Choose $\alpha$ (the Type I error rate you accept).
Check conditions for the chosen test.
Compute the test statistic and its p-value.
Decide: if $p \le \alpha$ , reject $H_0$ ("statistically significant evidence against $H_0$ "); otherwise fail to reject $H_0$ ("the data are consistent with $H_0$ ; not enough evidence against it").
Report in context: estimate, confidence interval, effect size, p-value, and what it means for the decision at hand.

Failing to reject is not evidence that $H_0$ is true: a small or noisy sample can fail to detect a real effect.

Common mistakes

Writing hypotheses about statistics (" $H_0: \xbar = 30$ ").
Choosing a one-sided alternative after seeing which way the data point.
Saying "we accept $H_0$ " or "the test proves there is no difference".
Treating "significant" as "important": with large samples tiny effects are significant. Look at the effect size and interval.
Running many tests and reporting only the significant ones (see multiple comparisons).

Example

A courier claims a mean delivery time of 12 minutes. Eight timed deliveries: 12, 15, 11, 14, 13, 16, 12, 15 minutes. $H_0: \mu = 12$ vs $H_1: \mu \ne 12$ , $\alpha = 0.05$ . A one-sample t-test gives $\xbar = 13.5$ , $t = 2.39$ on 7 df, $p = 0.048$ . Since $p \le 0.05$ , reject $H_0$ : the data are evidence that the mean differs from 12 minutes; the 95% CI (12.02 to 14.98 minutes) suggests deliveries take longer on average, by somewhere between a few seconds and about 3 minutes.