The practice
Mobile A/B Test Analytics, without the theatre.
This page is our working definition — the one we teach against. If your team uses the phrase for dashboard decoration, start here before you enrol.
What the phrase actually covers
Mobile A/B Test Analytics is the craft of assigning users (or devices, or accounts) to variants on iOS and Android, measuring what happened, and deciding whether a release should change. It includes store listings, in-app flows, messaging opt-ins, and paywalls. It does not include every multivariate toy, every personalisation engine, or every “we toggled a colour on Friday.”
The analytics half is not “more charts.” It is choosing a primary metric that the randomisation can actually move, naming guardrails that can veto a ship, and writing the uncertainty in plain English.
Four constraints web A/B folklore ignores
Store review is a calendar. You cannot always ship a fix tomorrow. Sample plans that assume instant rollback are fiction on iOS listing tests and on many in-app binary changes.
Identity is messy. Family sharing, device resets, and logout/login loops will corrupt naive user-level assignment. Thailand’s shared-device patterns make this worse than most North American tutorials admit.
SDKs drop events. Background kill, consent prompts, and offline queues mean your denominator is not the population you think it is. Instrumentation is part of experiment design, not a separate “data team” chore.
Guardrails have teeth. A conversion lift that coincides with a crash-rate bump is not a win. We treat stability and support load as first-class vetoes.
How Systemtracebase teaches it
We start from the release train and work backwards into statistics, not the other way around. The flagship Experiment Design for Mobile Releases is the spine. Shorter desks cover wiring, power, variant craft, and readout. If you only have time for one artefact, write the one-page memo: decision, uncertainty, what will not change.