Track 10 · Comparing Groups and Categorical Data · Lesson 11 of 11
Choosing a test
Choose an appropriate test for comparing groups from the study design, the type of outcome, the number of groups and the plausibility of the assumptions — and justify the choice.
30 minAdvanced7 graded exercisesMembers
Statistical software will run any test on any columns you give it. Whether the answer means anything depends on decisions made before clicking: are the groups independent or paired, is the outcome a count, a rating or a measurement, how many groups are there, and what do the data look like?
Choosing a test is mostly about the design, not the data. Four questions settle almost every case in this track:
- What is the unit, and are observations independent? Different people in each group (independent), the same people measured repeatedly (paired / repeated measures), or clustered (many weeks per store)?
- What type is the outcome? Categorical (counts in categories), ordinal (ratings), or numeric (measurements)?
- How many groups or conditions, and how many factors?
- Are the assumptions plausible? Shape (skew, outliers), spread (equal variances), sample size, expected counts.
Included with a Formula Quest membership
It has 7 graded exercises with feedback, hints and worked solutions. After it you will be able to:
- Choose a test from the unit of analysis, the dependence structure, the outcome type and the number of groups
- Decide between parametric and rank-based methods from shape, spread and sample size — before looking at p-values
- Recognise common flawed analyses: unadjusted multiple tests, chi-square on percentages, ignoring pairing, pseudo-replication
- Justify and report the chosen test with an effect size and the limits of the design