How to Tell Whether a Number Actually Moved

The checks that come before the explanation

A number moves, and within the hour there is a story about why. The story is usually plausible, sometimes correct, and almost never tested against the duller possibilities: that the period is not finished, that something in the pipeline changed, that the calendar already explains it, that the mix underneath shifted, or that this figure bounces about this much every week and nobody had looked.

What follows is a short sequence of checks to run before a meeting is spent on causes. It is not about designing an experiment, and not about judging whether an AI feature or an agent is doing its job — separate questions with their own methods. This one starts where most weeks start: a chart in front of you and a figure different from last time.

Ask whether the period is finished

The most common phantom decline is an incomplete bucket. Events arrive late — a queue retried overnight, a partner that reports weekly, refunds and settlements landing days after the sale, an offline client syncing when it reconnects. The most recent day, week or month is therefore always the lightest, and a chart read on Monday morning shows a fall that has filled itself in by Wednesday.

So ask first how long this particular figure takes to settle, then compare like with like: a complete week against a complete week, or a partial period against the same point inside previous periods rather than against their totals. If nobody knows the settling time, establishing it once is worth an hour: it retires a false alarm that would otherwise recur every month.

Ask what changed in the pipeline

Nothing changed is a claim, not a fact, and the tracking code is only one place a number can change without anybody editing it on purpose. A release moved a button, so an event now fires somewhere else. A filter that excluded internal traffic stopped matching after a domain change. A field was renamed upstream, arrives empty, and a whole segment quietly collapsed into unknown. Automated traffic arrived and nothing classifies it. Sampling was switched on as volume grew.

Check the deploy log, the pipeline's own run history and any recent edit to the metric's definition before hunting for a business cause. The tell is the shape: real behaviour tends to move like a slope or a wave, while a figure that jumps on a single boundary and holds its new level flat is far more often a collection or definition change than anything customers did.

Ask whether the calendar already explains it

A great deal of movement is just the calendar. Weekdays and weekends behave differently for nearly every product, so a week containing a public holiday is not comparable with the one before it, and months differ in how many working days they hold. Billing cycles concentrate renewals and cancellations on particular dates, and industry-wide quiet weeks move entire categories at once, none of it about you.

Compare against the same period a year earlier as well as against the previous one, and where there is enough history, look at the same week across several years before treating this year's dip as news. Guard against the opposite error too: a seasonal pattern is something you can show, and it is always slow at this time of year is a hypothesis to check, not a conclusion.

Ask whether the mix changed

A total can move while every part of it stays flat. If a fast-growing channel brings people who convert at a lower rate than everyone else, the blended conversion rate falls even though nothing got worse for anybody — and the same arithmetic runs in reverse, flattering a figure whenever a strong segment grows. Any average over a changing population moves for reasons that have nothing to do with quality.

The check is to split the figure by the two or three dimensions most likely to be shifting — channel, plan, region, device, customer age — and read the rates and the weights together. When every segment moves one way and the total moves the other, the mix is the answer, and the honest report is the segments alongside their sizes rather than the headline on its own.

Ask whether it is bigger than the usual bounce

Every number has a normal amount of movement, and most teams have never measured their own. Look back over a recent stretch when, as far as anyone knows, nothing was happening, and note how much the figure wandered from week to week anyway. That range is the bar this week's movement has to clear before it means anything.

Small denominators are where this bites hardest. A rate computed over a few dozen cases swings several points on one or two events, which is why weekly figures for a narrow segment look dramatic and then reverse themselves. Rare outcomes and averages dragged by a handful of large accounts behave the same way. When the denominator is small, either widen the window or publish the count beside the rate, so a reader can see what the percentage is made of.

Only then ask what caused it

A movement that survives those checks is real, which is not the same as explained. Several things happened in the same window — a campaign, a pricing change, a competitor's outage, a mention somewhere, a release nobody connected to this metric — and picking between them by which team happened to ship is the weakest evidence available.

The strongest evidence you are likely to have is a group that did not receive the change: a staged rollout where the remainder acts as a control, a region held back, accounts kept out of a campaign. Where none exists, say so in the sentence. Revenue rose after the launch is a fact; the launch raised revenue is a claim requiring a comparison you did not make. Writing the weaker sentence costs nothing and stops a false lesson being applied for a year.

Write down what you concluded

Whatever the answer, record it beside the metric: the date, the movement, what was checked, what it proved to be. The next person to notice the same dip is often the same person some months later, and the alternative to a note is running the whole sequence again from memory.

The notes also accumulate into something more useful than any one of them: a picture of how this figure normally behaves, which causes recur, and which confident explanations were later shown to be wrong. Teams that keep them stop having the same argument every quarter and get faster at spotting the movements that genuinely are new.