Reading conflicting segment results without cherry-picking
Segment surprises are normal. They become harmful when teams hunt for the slice that flatters the preferred variant and ignore the rest.
Start with the overall result on the pre-registered primary metric. That is the story the test was powered to tell. Segments are context, not a second lottery ticket.
When a segment flips the outcome, ask whether the segment was large enough to trust, whether assignment was balanced inside it, and whether the product can actually ship a different experience to that group. A beautiful insight that cannot be operationalized still matters for learning — it should not pretend to be a global ship decision.
Write segment findings in plain language: who saw the change, what moved, how big the group is, and what you will do. Avoid stacking five charts that each tell a different heroic narrative.
If you repeatedly need the same cut — new versus returning, iOS versus Android, Bangkok versus upcountry — promote that cut into the next test’s design. Planned comparisons beat post-hoc fishing.
segments experiment readout