21 Nov 2025 · Holdouts
Holdout groups that do not quietly hurt retention
A long holdout is how you learn whether the new onboarding still matters six months later. It is also how you accidentally keep 8% of users on a worse first-run while marketing celebrates a blended retention number. Mobile A/B Test Analytics has to hold both facts at once.
Size the holdout to the question, not the politics
If you only need to detect a large regression, the holdout can be small. If you want to publish a yearly “platform lift,” you need more. Marketing will always want the holdout smaller. Write the detectable regression in the brief before that conversation, or you will lose it.
Expiry is a feature
Holdouts without an end date become folklore. We date them. When the date hits, you either graduate the variant, kill it, or explicitly renew with a new memo. Silent extensions are how control users become a forgotten caste.
Watch retention on the holdout itself
If holdout retention falls while variant retention is flat-to-up, you may be looking at deprivation, not product quality. That is still information — but it is a different claim than “the new onboarding works.” Name it as deprivation in the readout. Alumni in the reading circle will call you on it if you do not.