Metric roles: decision metric, guardrails and revenue quality

• Framework · Measurement

  Activation moments and the aha threshold

  setup, aha and habit moments

  Work backward from the habit moment to the aha moment (first core value) to the setup moment (what must happen first), per account for team products. Find the product's own threshold by comparing retained and churned users across early actions, weighing coverage and precision, then confirm it with a randomized onboarding change.

• Framework · MeasurementARobustEvidence grade A. Robust evidence: replicated, peer-reviewed research.

  NPS and better-targeted survey metrics

  satisfaction and effort surveys

  NPS is qualified, not discredited: it predicts growth about as well as other satisfaction measures, but it throws away information, surveys only current customers and is not diagnostic. Pair any attitude score with behavior, such as actual referrals, and use targeted instruments: effort score, SUS, single-ease and worth-your-time questions.

• Framework · Measurement

  Using benchmarks honestly

  benchmark bands with definitions

  Benchmarks orient; they are not targets. They appear only as bands (below median, median, top quartile, top decile) with source, date, confidence and a definition-match flag. When definitions don't match, the product is compared with its own best cohort instead. Practitioner "good / great" labels map to median / top quartile.

• Framework · MeasurementARobustEvidence grade A. Robust evidence: replicated, peer-reviewed research.

  Experiment hygiene: trust checks before reading results

  trustworthy A/B testing checklist

  Before trusting a result: check the sample ratio, run A/A tests, treat surprising wins as likely bugs, never stop a fixed-horizon test at the first p < 0.05, and run at least one full week (ideally two) to cover weekly cycles and novelty or change-aversion effects. Pre-register the metric, the horizon and the decision rule.

• Framework · Measurement

  Experiment win rates and the winner's curse

  how often tests really win

  In mature organizations about 1 in 10 experiments wins, and at that base rate a significant result has roughly a 22% chance of being a false positive. At low power any significant result overstates the true effect. Replicate surprising wins, report intervals and show a shrunk estimate next to the raw one.

• Framework · Measurement

  Growth accounting and the user state model

  where active users come from and go

  Split active users into new, retained, resurrected and churned each period; the user quick ratio, (new + resurrected) / churned, must exceed 1 to grow. Duolingo's state model classifies every user daily and showed that lifting current-user retention (CURR) moved daily actives most. It is the tool for splitting a decline by phase.

Sources

• Influence · Robert Cialdini

  Ch.3, p.76

• Risk-aware product decisions in A/B tests with multiple metrics · Schultzberg, Ankargren and Franberg (Spotify), 2024

  ARobustEvidence grade A. Robust evidence: replicated, peer-reviewed research.

• Email Volume Optimization at LinkedIn · Gupta et al., KDD 2016, 2016

  ARobustEvidence grade A. Robust evidence: replicated, peer-reviewed research.

Source IDs: INF-K-03 · F26

Book content is paraphrased, never quoted at length. Last verified October 24, 2026.