A/B Testing

An A/B test compares two versions of an experience by randomly assigning users to one of them and measuring a predefined metric. Randomisation is what makes the comparison causal: because assignment is independent of who the user is, any systematic difference in outcome is attributable to the change rather than to the two groups being different to begin with. That is the whole value, and it is also what before-and-after comparisons cannot give you, since seasonality, marketing activity and product changes all move numbers on their own. Running one honestly requires a few commitments made before launch. Pick a single primary metric and the minimum effect worth detecting, and use those to compute how long the test must run — a test whose duration is decided after the data comes in is a search for a favourable stopping point. Run in whole business cycles, usually complete weeks, since weekday and weekend traffic behave differently. Decide up front which secondary metrics act as guardrails, so a lift in conversion that comes with a rise in refunds is not recorded as a win. And check that assignment actually worked: unequal group sizes are the usual sign of an instrumentation bug that invalidates the result. The main practical limit is traffic. Most changes produce small effects, and small effects need large samples, so low-traffic products often cannot resolve the differences they care about — in which case sequencing changes, testing bigger swings, or relying on qualitative evidence is more honest than running an underpowered test and reading the result.