Framework · Measurement
Experiment win rates and the winner's curse
In mature organizations about 1 in 10 experiments wins, and at that base rate a significant result has roughly a 22% chance of being a false positive. At low power any significant result overstates the true effect. Replicate surprising wins, report intervals and show a shrunk estimate next to the raw one.
Plain label: how often tests really winAlso known as: base rate, false positive risk, winner's curse, P(win)
- Microsoft: roughly a third of ideas positive, a third flat, a third negative. Bing, Booking, Google Ads and Netflix report about 10-15% wins; Airbnb search relevance about 8% (benchmark confidence A).
- The median success rate across large organizations is about 10% (benchmark confidence A).
How it plugs in. P(win) on each card starts from the base rate with bounded multipliers and is clamped to 5-35%, labeled an uncalibrated prior until a technique family has at least 50 logged outcomes. Underpowered wins report a shrunk estimate weighted toward the prior range, and the holdback re-read decides the number that feeds local priors.
Sources
A/B-test power, base rates of success and replication · Kohavi et al., Econ Journal Watch 23(1), 2026
Evidence grade A. Robust evidence: replicated, peer-reviewed research.Trustworthy Online Controlled Experiments · Kohavi, Tang and Xu, Cambridge University Press, 2020
Evidence grade A. Robust evidence: replicated, peer-reviewed research.
Source IDs: F26
Book content is paraphrased, never quoted at length. Last verified October 24, 2026.
Your product
Find the stage your product is stuck in
Gamify It reads your product, picks the one move that matters at its stage, and sizes the test to your traffic. Connecting is free; a Pro subscription starts it.