Framework · Measurement
Sample size, minimum detectable effect and weeks to read
Per arm, n is about 16 x p(1 - p) / d^2 at alpha 0.05 and 80% power, where d is the absolute change to detect. Weeks to read add enrollment, metric maturation and any app-release lag. A step is Ready at 6 weeks or less, Slow at 6-10 and Underpowered beyond 10, where bigger bundles, proxies, triggered analysis or the low-traffic loop take over.
Plain label: can this step be tested in timeAlso known as: MDE, power analysis, weeks to read, detectability, CUPED
Worked examples (alpha 0.05, 80% power, relative lifts). A 5% baseline needs roughly 30,000 users per arm to detect a 10% lift and over 120,000 for a 5% lift. A 40% baseline needs about 600 per arm for a 20% lift. Halving the lift you want to detect roughly quadruples the sample.
Maturation. A 7-day activation metric adds a week; a 60-day repeat-purchase metric adds about 8.6 weeks.
Small products. Design for 10-20% MDEs on single changes, or a larger one for a bundle on an observed broken step; use higher-baseline metrics earlier in the funnel; analyze only users who reach the changed step; reduce variance with pre-period data (CUPED cut variance about 50% at Bing Evidence grade A. Robust evidence: replicated, peer-reviewed research.).
High stakes. Pricing, paywall and billing tests use alpha 0.01.
Sources
Trustworthy Online Controlled Experiments · Kohavi, Tang and Xu, Cambridge University Press, 2020
Evidence grade A. Robust evidence: replicated, peer-reviewed research.Improving the sensitivity of online controlled experiments by utilizing pre-experiment data (CUPED) · Deng, Xu, Kohavi and Walker, WSDM 2013, 2013
Evidence grade A. Robust evidence: replicated, peer-reviewed research.
Source IDs: F26
Book content is paraphrased, never quoted at length. Last verified October 24, 2026.
Your product
Find the stage your product is stuck in
Gamify It reads your product, picks the one move that matters at its stage, and sizes the test to your traffic. Connecting is free; a Pro subscription starts it.