Case · M4 × Pay / Stay ("why we measure")
Google: more ads taught users to ignore ads
Google ran long-running experiments on search ads and found users learn: after seeing low-quality ads they become less willing to click any ads. The finding led Google to cut ad load on mobile search by 50% and give ad quality more weight in the auction (KDD 2015).
The outcome · 2015
Long-running experiments showed users' tendency to click ads shifts with the quality of ads they have seen (ad blindness or sightedness). The results led to a 50% cut in ad load on Google's mobile search and an auction change favoring ad quality (Hohnhold, O'Brien & Tang, KDD 2015).
Stage cell: M4 × Pay / Stay ("why we measure").
What they did. Held-out user groups saw different ad loads and ad quality over long periods, so the team could measure how users' propensity to engage with ads changed, not just this week's clicks.
The mechanism. People learn from repeated exposure: low-quality ads teach users that ads aren't worth a look. Only long-term measurement F26 catches that learning, and realistic effect sizes come from controlled tests, not intuition F23 (Effects of feedback interventions on performance, Kluger & DeNisi, Psychological Bulletin, 1996).
The outcome. -50% mobile ad load (2015) Case grade A. From a regulator, a public filing or a peer-reviewed study., plus an auction change that weighted ad quality more heavily.
What it means at your stage. At M4-M5, every monetization or notification lever needs a long-running holdout. Pick a guardrail metric that captures learned avoidance (click propensity, opt-outs, uninstalls) and watch it for weeks, not days.
Caveats. Company authors reporting on their own product; mechanism link Evidence grade B. Moderate evidence: solid studies with some limits..
Would you have called it?
This case is also a round of Predict the Outcome: read the situation, pick what you think happened, then see the evidence.
Play this roundTry this
The techniques behind it
Each one is a starting point, not a recipe. Gamify It turns a technique into an experiment card for your product: the change, the metric, the guardrail and how long the test needs at your traffic.
- TechniqueEvidence grade B. Moderate evidence: solid studies with some limits.
Test notifications against a holdout before scaling
notification holdout test
Before adding or cutting a message, send it to most users and hold out a random group. Compare sessions, opt-outs, uninstalls and organic shares, so one good or bad anecdote doesn't set the cadence.
Limits
What we don’t claim
- Habit metrics need the longer habit horizon (weeks to months) before they can be read.
- No success is attributed to a single technique: real outcomes have many causes.
- Company names are used editorially; no endorsement is implied.
Sources
Focus on the Long-Term: It's Better for Users and Business · KDD 2015 (Hohnhold, O'Brien & Tang, Google), 2015
Case grade A. From a regulator, a public filing or a peer-reviewed study.Evidence grade B. Moderate evidence: solid studies with some limits.Effects of feedback interventions on performance · Kluger & DeNisi, Psychological Bulletin, 1996
Evidence grade A. Robust evidence: replicated, peer-reviewed research.
Book content is paraphrased, never quoted at length. Last verified October 24, 2026.
Your product
Find the stage your product is stuck in
Your product has its own version of this story. Find the stage it’s in and the move that fits it. Connecting is free; a Pro subscription starts it.