Customer conversion experiment: ship the new checkout?

Nakhla Market (a fictional online retailer) has redesigned its checkout into a single page (variant B). The current multi-step checkout is variant A. New visitors were randomly assigned to A or B for 14 days, and the decision meeting is next week.

Stakeholders and what they want

• The head of product built B and wants to ship it before the peak season.
• The chief financial officer wants to know what B is worth in riyals per month — and what it could cost if it is actually worse.
• The engineering lead is worried that B's heavier page loads more slowly, which could hurt customers in ways a 14-day test does not capture.
• The experimentation council (you are its analyst) insists that the metric, the effect worth detecting and the sample size are fixed before the data are read, and that every check is reported, not only the favourable ones.

Constraints. The test window was capped at 14 days by a code freeze, which gave about 16,000 users in total. For this scenario, assume the business plans with 35,000 new visitors reaching checkout per month, a gross margin of 25% on order value, and a pre-registered guardrail: B must not make the median checkout page load more than 3% slower.

The data. checkout_ab has one row per user: the day they entered the test, variant, device, location segment, whether they bought (converted, 0/1), order value (revenue_sar, 0 for non-buyers) and the checkout page-load time. The dataset is synthetic: it was generated to behave like a real e-commerce experiment, and its numbers are not evidence about any real website.

Your deliverable is a decision memo for the product review: what you planned, what you checked, what you found, how sure you are, and what Nakhla Market should do next.