Framework · Measurement
The low-traffic loop (directional reads)
When a step can't be read within about 10 weeks, use designs built for small samples: a pre-registered before/after with a concurrent holdback, a sequential test that can stop early, a "probability it helped" read under about 200 users per arm, or a structured 5-user qualitative readout. Decisions are Keep, Revert or Inconclusive.
Plain label: testing with small samplesAlso known as: directional read, before/after with holdback, probability it helped, sequential test
| Design | How it works | Label |
|---|---|---|
| Before/after with a holdback | A window fixed in advance (for example 6-8 weeks), with a 10-20% holdback where the change allows | Directional |
| Sequential | Always-valid or group-sequential bounds; early stops overstate effects | Causal when it stops on a bound |
| Probability it helped | Flat-prior probability that the variant beat control, with a wide interval | Directional |
| Qualitative readout | 5 users on the changed step, or 5 conversations, logged and graded | Learning |
Decision rule, set in advance. Keep if the probability it helped is at least 0.9 and no guardrail shows harm; Revert if it is below 0.5; otherwise Inconclusive, and a low-risk change may stay with the holdback running.
Why. At about 30 sign-ups a week, a bundled change aimed at a 40% relative lift on a 32% baseline takes about 15 weeks to read as an A/B test, and a 20% MDE takes over a year. Small teams make big changes and read them honestly.
What it feeds. Directional reads update local priors at reduced weight and never count toward win-rate calibration.
Sources
Trustworthy Online Controlled Experiments · Kohavi, Tang and Xu, Cambridge University Press, 2020
Evidence grade A. Robust evidence: replicated, peer-reviewed research.Why you only need to test with 5 users · Nielsen Norman Group (Jakob Nielsen), 2000
Evidence grade B. Moderate evidence: solid studies with some limits.
Book content is paraphrased, never quoted at length. Last verified October 24, 2026.
Your product
Find the stage your product is stuck in
Gamify It reads your product, picks the one move that matters at its stage, and sizes the test to your traffic. Connecting is free; a Pro subscription starts it.