Skip to content
Gamify It.
Connect

Technique · Hooked

Test notifications against a holdout before scaling

Before adding or cutting a message, send it to most users and hold out a random group. Compare sessions, opt-outs, uninstalls and organic shares, so one good or bad anecdote doesn't set the cadence.

Evidence grade B. Moderate evidence: solid studies with some limits.Return

Plain label: notification holdout testAlso known as: holdout, notification test, send-time test, uninstall risk, message experiment

How it works

Teams tend to either fear every message or trust every message. A randomized holdout shows a message's net effect, including the hidden cost of revoked permission, which open rates never reveal.

What the research says

  • A reading app expected a holiday greeting to cause uninstalls; users shared it widely instead (company-reported).

    Hooked Ch.7 (YouVersion), 2013

Before you start

  • Measurement readiness L3 or higher.
  • Enough weekly recipients to read a holdout; otherwise run a directional before-after read

How to build it

Mobile
Hold a random slice of eligible users out of a new push type for several weeks and compare sessions, disables and uninstalls.
Email and SMS
Randomly suppress a new lifecycle email for a holdout and compare 30-day retention and unsubscribes.
B2B
Randomize at the account level so teammates share a condition.

What can go wrong

  • Reading only open rates.
  • Stopping on the first day that looks significant.
  • Randomizing by user when a team shares the channel.

Pairs well with

  • Attribute sessions with the trigger inventory.
  • Re-read after 8-12 weeks for durability.

In practice

Where it shows up

  • Duolingo · 2020

    Round

    Duolingo: smarter reminders, not more reminders

    The bandit algorithm lifted total DAU 0.5% and new-user retention 2% over a strong baseline, without sending more notifications (Yancey & Settles, KDD 2020; company authors, peer-reviewed).

    M4 ×Return
    Case grade for the number: A. Evidence grade for the mechanism link: B.

  • Google · 2015

    Round

    Google: more ads taught users to ignore ads

    Long-running experiments showed users' tendency to click ads shifts with the quality of ads they have seen (ad blindness or sightedness). The results led to a 50% cut in ad load on Google's mobile search and an auction change favoring ad quality (Hohnhold, O'Brien & Tang, KDD 2015).

    M4 ×StayPay
    Case grade for the number: A. Evidence grade for the mechanism link: B.

Limits

What we don’t claim

  • The Hook Model is a practical design tool, not a validated or complete theory of behavior.Hooked, Introduction

Sources

  • Hooked · Nir Eyal

    Ch.7

Source IDs: HK-TRG37

Book content is paraphrased, never quoted at length. Last verified October 24, 2026.

Your product

Find the stage your product is stuck in

Gamify It reads your product, picks the one move that matters at its stage, and sizes the test to your traffic. Connecting is free; a Pro subscription starts it.

Add Gamify It to your assistantTake the 2-minute check-up