Track 8 · Hypothesis Testing · Lesson 10 of 14
Power and sample size
Explain what determines the power of a test, compute power for simple cases, and choose a sample size that gives adequate power before collecting data.
The checkout experiment and the independent-samples delivery study both ended “not significant”, although the synthetic data were built with real effects. Neither study was large enough to have a good chance of detecting them. Power analysis is how you find that out before spending two weeks of traffic or a month of deliveries.
Power is the probability that a test rejects H0 when a specific alternative is true — the chance of detecting an effect of a given size. It depends on four things:
- Effect size: bigger true effects are easier to detect.
- Variability (σ): noisier data hide effects.
- Sample size (n): the standard error shrinks like .
- Significance level (α) and sidedness: a stricter α, or a two-sided test, makes rejection harder.
By convention studies aim for power of at least 0.80 (often 0.90). Fixing three of the quantities determines the fourth — most often we fix the effect worth detecting, σ, α and the target power, and solve for n.
Included with a Formula Quest membership
It has 8 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Name the four drivers of power: effect size, variability, sample size and α
- Compute the power of a z-test and the sample size for a target power
- Use a power calculator for t-tests and two-proportion tests before collecting data
- Explain why post-hoc “observed power” is uninformative and the confidence interval is not