Type I and Type II errors, significance level and power

A test can wrongly reject a true null (Type I, probability α) or miss a real effect (Type II, probability β); power = 1 − β is the chance of detecting an effect of a given size.

\alpha = P(\text{reject } H_0 \given H_0 \text{ true}), \qquad \beta = P(\text{fail to reject } H_0 \given H_1 \text{ true}), \qquad \text{power} = 1 - \beta

$\alpha$: Significance level = Type I error rate.
$\beta$: Type II error rate, for one specific effect size.

\text{power} = P(T > t_{1-\alpha/2,\,\nu};\ \lambda) + P(T < -t_{1-\alpha/2,\,\nu};\ \lambda)

Two-sided t-test power from the noncentral t distribution (both rejection regions counted, as the engine does).

$T$: A noncentral t variable with $\nu$ df and noncentrality $\lambda$ .
$\nu$: Degrees of freedom: $n - 1$ (one-sample, paired) or $n_1 + n_2 - 2$ (two-sample).
$\lambda$: Noncentrality: $d\sqrt{n}$ (one-sample/paired) or $d\sqrt{n_1 n_2/(n_1+n_2)}$ (two-sample).
$d$: The standardised effect size (Cohen's d) you want to detect.

When to use

Before a study: choose $\alpha$ , the smallest effect worth detecting, and the target power (often 0.80 or 0.90), then solve for the sample size.
After a non-significant result: ask whether the study had enough power to detect a meaningful effect (using a pre-specified effect size, not the observed one).

Assumptions

Power calculations assume the test's model (here normal data with equal variances for the two-sample t) and a guessed effect size and SD. They are planning tools, as good as those guesses.

How to interpret

	$H_0$ true	$H_0$ false
Reject $H_0$	Type I error ( $\alpha$ )	Correct (power)
Fail to reject	Correct ( $1 - \alpha$ )	Type II error ( $\beta$ )

Power increases with: a larger true effect, a larger sample, less noise (smaller SD, pairing, better measurement), a larger $\alpha$ , and a one-sided test (in the predicted direction). Lowering $\alpha$ reduces Type I errors but, at a fixed n, raises $\beta$ .

Common mistakes

Thinking $\alpha$ is the probability that a significant result is a false positive (that depends on how often $H_0$ is true and on power).
"Post-hoc power" computed from the observed effect: it is just a transformation of the p-value and adds nothing.
Treating a non-significant result from an underpowered study as evidence of no effect.
Forgetting that power refers to one specific effect size.

Example

Two groups, two-sided t-test, $\alpha = 0.05$ , true effect $d = 0.5$ . With 30 per group, power = 0.478 — more likely to miss the effect than to find it. Reaching 80% power needs 64 per group (achieved power 0.801). A one-sample design with $n = 30$ and $d = 0.5$ has power 0.754; it needs $n = 34$ for 80%.