Experiments and A/B Testing skill check

Skills covered

Explain why randomised experiments support causal claims · Choose the randomisation and analysis unit · Choose primary, secondary and guardrail metrics · Estimate a conversion rate with its uncertainty · Plan sample size and the minimum detectable effect · Validate an experiment with A/A tests and sample ratio checks · Analyse an A/B test: difference, interval, test and lift · Explain why peeking inflates false positives · Handle multiple variants and multiple comparisons · Analyse segments without fishing · Check guardrail metrics and weigh trade-offs · Describe sequential designs and alpha spending · Decide under uncertainty with costs and practical significance · Write a justified experiment recommendation · Spot common experimentation mistakes

Covers Track 11: Experiments and A/B Testing. Passing it, together with completing every lesson, earns the track certificate.