Run a holdout, live
Set the size of the test, how much to hold back, the baseline conversion rate and the lift you expect. Watch the confidence band tighten week by week, and see whether the result is one you can actually call a win.
Test design
Total customers in the test. 18,000 treated, 2,000 held back.
Share held back as the control group. Bigger holdout, tighter reading, less reach.
Weekly conversion rate of a control customer with no campaign.
How much better the treated group is assumed to convert, in relative terms.
Weeks of accrual. More weeks, more sample, a narrower interval.
The read
The whole confidence interval sits above zero. Even the pessimistic read shows real lift.
Week. The band narrows as the sample grows; a dot turns green once that week is significant.
Illustrative and deterministic: the series is simulated from the inputs, so the same design always reads the same way. In production the holdout runs on live customers and the interval is measured, not assumed.