Sample size myths that quietly ruin app experiments
Large user counts feel reassuring. They are not, by themselves, proof that an A/B comparison is ready. What matters is how many users were correctly assigned, exposed to the experience, and counted against the same success definition.
A common myth is that any difference that appears after a few thousand sessions is “real.” Session counts can inflate exposure when the same person returns repeatedly. Assignment should usually sit at the user or account level for most app product questions, with sessions used only when the hypothesis truly concerns a single visit.
Another myth: equal traffic shares guarantee fairness. If one variant loads slower, or if a bug drops users from a funnel step, the comparison is tilted even when the split looks fifty-fifty. Check completion of the first meaningful screen for each arm before celebrating a conversion lift.
Rare events need patience. Purchase or subscription conversion in a niche app may require weeks, not days. Shortening the window to meet a launch date does not create information — it creates confidence theater.
Before launch, write down the expected baseline rate and the smallest lift that would change the product plan. That pair anchors a rough sample target. If the calendar cannot support it, shrink the scope of the question instead of pretending a thin sample will carry a heavy decision.
sample size app analytics