Statistics Quest

Resampling and Statistical Reasoning

Simulation-based inference and healthy scepticism.

Bootstrapping, permutation tests and Monte Carlo simulation; robust statistics and outliers; missing data; sensitivity analysis; Simpson's paradox, selection and survivorship bias, p-hacking, publication bias, reproducibility and ethics.

Lessons

  1. 1The bootstrapUse bootstrap resampling to estimate a standard error and a confidence interval, and recognise the situations in which the bootstrap fails.35 minFree
  2. 2Permutation testsRun and interpret a permutation test by shuffling group labels, and distinguish exact from Monte Carlo permutation p-values.35 minMembers
  3. 3Monte Carlo simulationAnswer probability and method-checking questions by simulation, and quantify and control the Monte Carlo error of the answer.30 minMembers
  4. 4Robust statisticsMeasure how sensitive a statistic is to outliers, use robust alternatives (median, trimmed mean, MAD), and decide when a non-robust statistic is still the right one.35 minMembers
  5. 5Missing dataClassify missing-data mechanisms (MCAR, MAR, MNAR), explain when complete-case analysis and simple imputation give biased answers, and report missingness honestly.35 minMembers
  6. 6Sensitivity analysisTest whether a conclusion survives other defensible analysis choices and assumptions, find tipping points, and report fragile results honestly.35 minMembers
  7. 7Simpson's paradoxRecognise when an aggregated comparison reverses within subgroups, explain the reversal through a confounder, and choose the comparison that answers the question.30 minMembers
  8. 8Selection and survivorship biasRecognise how the way units enter a dataset — surviving, volunteering, being selected or staying in a study — can bias estimates and create spurious relationships, and quantify simple cases.30 minMembers
  9. 9p-hacking and publication biasExplain how multiple testing, flexible analysis and selective publication produce false or exaggerated findings, and apply corrections and safeguards such as adjustment and pre-registration.35 minMembers
  10. 10Reproducibility and ethicsMake an analysis reproducible, report results honestly and in proportion, and protect the privacy of the people behind the data.30 minMembers