Simpson's Paradox

Simpson's paradox is the situation where a trend holds in every segment of your data and reverses when the segments are pooled. Conversion improved for self-serve signups and improved for enterprise signups, and the blended conversion rate fell, because self-serve grew much faster over the same period and self-serve converts at a lower rate. Nothing in that sentence is a mistake; both readings are arithmetically correct. What changed the headline number was the composition of the population, not the behaviour of anyone in it. This is worth naming because the everyday version is far more common than the textbook reversal. Most of the time the mix simply shifts enough to move the aggregate, and the movement gets attributed to a product change or a campaign that had nothing to do with it. Average revenue per account falls after a successful month of small-plan acquisition. Support satisfaction drops after expanding into a market with different expectations. Latency rises after onboarding one large customer whose workloads are heavier. Each looks like a regression and is a change in who is being averaged. The habits that catch it are cheap. Before explaining a movement in a blended number, plot the same metric split by the dimensions most likely to have changed weight — plan, channel, region, tenure, device — and check whether the segments moved at all. Report the mix alongside the metric, so a shift in the denominator is visible in the same chart rather than discovered in an argument. And treat any aggregate that combines populations you would never manage the same way as a reporting convenience rather than as a fact about the product. The reverse mistake is worth guarding against too: splitting a number far enough will eventually produce a segment that moved in the direction you hoped for, so the decision about which dimensions to check belongs before the data is pulled rather than after the first cut disappoints.