Experimentation and A/B Testing — final assessment

Skills covered

Check whether events are independent · Calculate conditional probabilities · Compute an expected value · Calculate the standard error of a proportion · Apply the central limit theorem · Distinguish parameters from statistics · Identify sources of sampling bias · Build an interval for a proportion · Build an interval for a difference in proportions · Build a Welch interval for a difference in means · Interpret a confidence interval correctly · Calculate and use a margin of error · Formulate null and alternative hypotheses · Compute and interpret a p-value · Plan power and sample size · Compute a test statistic · Control error rates across many tests · Choose an appropriate test · Explain why randomised experiments support causal claims · Choose the randomisation and analysis unit · Choose primary, secondary and guardrail metrics · Estimate a conversion rate with its uncertainty · Plan sample size and the minimum detectable effect · Validate an experiment with A/A tests and sample ratio checks · Analyse an A/B test: difference, interval, test and lift · Explain why peeking inflates false positives · Handle multiple variants and multiple comparisons · Analyse segments without fishing · Check guardrail metrics and weigh trade-offs · Describe sequential designs and alpha spending · Decide under uncertainty with costs and practical significance · Write a justified experiment recommendation · Spot common experimentation mistakes