Choosing a test: decision guide
Match the question, the outcome type and the study design to an appropriate method, then check its assumptions.
When to use
Before any analysis. Answer three questions: What is the question (compare groups, relate variables, estimate one quantity)? What type is the outcome (quantitative, binary, categorical, ordinal)? What is the design (one sample, two independent groups, paired, several groups)?
| Question / design | Quantitative outcome (roughly normal or large n) | Skewed / ordinal / small n | Binary or categorical outcome |
|---|---|---|---|
| One sample vs a benchmark | One-sample t | Wilcoxon signed-rank or sign test | One-proportion z / exact binomial; χ² goodness of fit |
| Two independent groups | Welch two-sample t | Mann–Whitney; permutation test | Two-proportion z; Fisher exact if counts are small |
| Paired / before–after | Paired t | Wilcoxon signed-rank | McNemar's test (paired binary) |
| Three or more groups | One-way ANOVA or Welch ANOVA, then Tukey | Kruskal–Wallis | χ² test of independence |
| Two categorical variables | — | — | χ² independence; Fisher for sparse 2×2 |
| Relationship between two quantitative variables | Pearson r, simple regression | Spearman / Kendall | — |
| Predict an outcome from several variables | Multiple regression | Transform or robust methods | Logistic regression |
| Any statistic, few assumptions | Bootstrap CI | Permutation test | Bootstrap |
Assumptions
Every row inherits its method's assumptions; independence and a sound design matter more than the choice between parametric and rank tests. Check them with plots first (Shapiro–Wilk and Levene/Brown–Forsythe only as supporting evidence).
How to interpret
- Prefer methods that give an estimate with an interval, not just a p-value.
- Parametric tests answer questions about means; rank tests about the tendency of one group to exceed another. Pick the one whose question you care about, not the one with the smaller p.
- With several tests, plan a multiple-comparison adjustment.
Common mistakes
- Trying several tests and reporting the one that "worked".
- Ignoring pairing or clustering (e.g. treating repeated visits by one customer as independent).
- Running multiple t-tests instead of ANOVA plus a post-hoc procedure.
- Using a chi-square test on percentages or means.
- Choosing a nonparametric test only because a normality test gave p < 0.05 with a large sample.
Example
A clinic records waiting times (strongly right-skewed, 15 patients per clinic) for three clinics. Question: do waiting times differ between clinics? Outcome: quantitative but skewed; design: three independent groups, small n → Kruskal–Wallis, followed by Holm-adjusted pairwise Mann–Whitney tests; report medians and IQRs per clinic. If the groups were large (say 200 each), Welch's ANOVA on the means would also be reasonable, and it answers a question about average waiting time, which may matter more for staffing.