Survivorship Bias

Survivorship bias is what happens when the population you measured is the one that stayed. It is the most ordinary error in product reporting and the hardest to see, because the missing accounts leave no row in the table you are looking at. A satisfaction score collected in-product is answered by people still using the product. A study of which onboarding steps predict success is run over accounts that finished onboarding. Feature usage looks healthy because the customers who found the feature useless stopped opening it and eventually stopped paying, taking their zeroes out of the average with them. In each case the number is arithmetically correct and describes a group that was selected by the very outcome you are trying to explain. The tell is that the conclusion always flatters: survivors report high satisfaction, survivors used the feature, survivors completed the flow. When a finding has no unhappy tail at all, the tail probably left rather than never existed. Correcting it is mostly a matter of insisting on the denominator you actually meant. Anchor the analysis on the cohort as it existed at the start — everyone who signed up in that month, including the accounts that are gone — rather than on who is present today, so churn appears as a result rather than as an absence. Ask where the leavers went in the same query that reports the stayers. Sample dissatisfaction outside the product, where the people who left can still be reached, since an in-app survey structurally cannot reach them. And be sceptical of any comparison between long-tenured and new accounts, because tenure is not a treatment: the old accounts are old partly because they were the ones the product suited. The same shape shows up in competitive research, where the vendors and features you can study are the ones that lasted, and in any internal retrospective that examines shipped projects without the ones that were cancelled halfway. Neither is a reason to stop looking; both are a reason to say out loud which population the finding describes.