Percentile

A percentile is the value below which a given share of your measurements fall: the median, written p50, is the value half of them are under, and p95 is the value ninety-five percent are under. Percentiles matter because averages hide the shape of a distribution, and for anything a user waits on, the shape is the experience. A page whose average response is comfortable can still be slow for a substantial minority of requests, and no amount of staring at the average will reveal it, because a small number of very slow responses barely move a mean computed over a large number of fast ones. The median tells you what a typical request looks like; the upper percentiles tell you what the worst part of the experience looks like and how many people are living in it. This applies well beyond latency. Time to first response in support, onboarding duration, the length of a generated answer, the cost of an individual request against a model — all of them have long tails, and in every case the average describes a request that may not exist. Two cautions are worth carrying. Percentiles do not add up or average across services, so combining the p95 of three components tells you very little about the p95 of the path through all three, and a per-service dashboard can look healthy while the end-to-end experience does not. And a percentile computed over a long window smooths away exactly the incident you were trying to find, so the window is part of the number rather than a display setting. Report the median and an upper percentile together, and name the window in the label, because a percentile quoted without one is not a measurement. The last thing worth saying is that percentiles describe requests rather than people. One user hitting the slow path repeatedly and a hundred users each hitting it once produce the same p99 and are very different problems, so a tail worth acting on usually needs a second look at who is inside it.