Rate Limit Backoff

Backoff is the retry strategy a client uses when an API says no — usually a 429 or a 503. Instead of retrying immediately, the client waits, and the wait grows with each attempt: one second, two, four, eight. Randomised jitter is added so that a thousand clients that failed together do not retry together. Model APIs make this unusually important because limits bind on two axes: requests per minute and tokens per minute. A workload can be well under the request limit and still be throttled by long prompts, which is confusing exactly when you are debugging under load. Three rules keep it sane. Respect the Retry-After header when the provider sends one — it is better information than your formula. Cap the total attempts and surface a real failure rather than retrying into a queue nobody drains. And make retries idempotent: a generation retried three times after a timeout has, from the provider's side, been billed three times. For user-facing paths, prefer a fast fallback over a long retry chain. Two seconds of stalling costs more goodwill than a slightly worse answer from a smaller model.