3 Apr 2026 · Store listings
When App Store listings skew the test you thought you ran in-app
A squad changes paywall copy inside the app and, the same week, ships new Play Store screenshots. Activation moves. They credit the paywall. They are often wrong. Listing creative changes who arrives. In-app copy changes what those arrivals do. Mixing the two is how Mobile A/B Test Analytics turns into folklore.
Two clocks, two populations
Store experiments update on Apple and Google’s clocks, not yours. In-app assignment can flip with the next binary. The people who saw the new screenshots are not a random slice of your existing user base; they are disproportionately new or returning from search. If your in-app test includes those new users without a flag, the paywall variant inherits a composition change.
Separate the memos
We ask Variant Craft students to keep two briefs: one for listing CTR and install quality, one for in-app conversion among users already assigned. If both must run in the same month, freeze listing creative for the in-app window, or exclude new installers from the in-app primary metric. Neither option is elegant. Both are cleaner than a blended dashboard.
Install quality is allowed to be the listing primary
Day-1 activation, not tap-through, is the listing metric we prefer when the product is regulated or has a heavy onboarding. A prettier screenshot that recruits tourists will look like a win on CTR and a bruise on activation. Our bank case note is the example we keep on the wall.