Framework · Measurement
Experiment hygiene: trust checks before reading results
Before trusting a result: check the sample ratio, run A/A tests, treat surprising wins as likely bugs, never stop a fixed-horizon test at the first p < 0.05, and run at least one full week (ideally two) to cover weekly cycles and novelty or change-aversion effects. Pre-register the metric, the horizon and the decision rule.
Plain label: trustworthy A/B testing checklistAlso known as: SRM, sample ratio mismatch, A/A test, Twyman's law, peeking, novelty effect
- Sample ratio mismatch. A chi-square check that the observed split matches the design. About 6% of Microsoft tests had one, and it usually invalidates the result Evidence grade A. Robust evidence: replicated, peer-reviewed research..
- A/A tests. About 5% should be significant at alpha 0.05; more means a broken pipeline.
- Twyman's law. A result that looks surprisingly good is probably a bug: re-check before celebrating.
- No peeking. Checking a fixed-horizon test repeatedly and stopping at the first significant read can multiply false positives about fivefold. Use a fixed horizon or a valid sequential method, and remember that early-stopped estimates run high Evidence grade A. Robust evidence: replicated, peer-reviewed research..
- Duration. At least one full weekly cycle, preferably two. Novelty bumps and change-aversion dips usually change the size of an effect, not its direction.
- Before the horizon, only exposure and the sample ratio are shown.
Sources
Trustworthy Online Controlled Experiments · Kohavi, Tang and Xu, Cambridge University Press, 2020
Evidence grade A. Robust evidence: replicated, peer-reviewed research.Diagnosing Sample Ratio Mismatch in online controlled experiments · Fabijan et al., KDD 2019, 2019
Evidence grade A. Robust evidence: replicated, peer-reviewed research.Peeking at A/B tests: why it matters, and what to do about it · Johari, Koomen, Pekelis and Walsh, KDD 2017, 2017
Evidence grade A. Robust evidence: replicated, peer-reviewed research.Evaluating for the Long Term: Learnings from Industry · Sigerson et al., arXiv, 2026
Evidence grade A. Robust evidence: replicated, peer-reviewed research.
Source IDs: F26
Book content is paraphrased, never quoted at length. Last verified October 24, 2026.
Your product
Find the stage your product is stuck in
Gamify It reads your product, picks the one move that matters at its stage, and sizes the test to your traffic. Connecting is free; a Pro subscription starts it.