Fallback Model

A fallback model is the secondary model your application routes to when the primary one cannot serve a request — it returned an error, it hit a rate limit, it timed out, or the provider is having an incident. It is the standard availability pattern for products built on hosted inference, because your own uptime commitment cannot be better than your provider's unless you have somewhere else to send traffic. Implementations range from a few lines of retry logic naming a second model, through a routing layer that fails over between providers, to a gateway that health-checks endpoints and shifts traffic before requests start failing. The design question that gets skipped is what a degraded answer is worth. A fallback to a smaller or cheaper model keeps the feature responding, but it responds differently: output format may drift, instruction-following weakens, and behaviour you tuned into the primary may not survive the switch. For a summarisation feature that is an acceptable trade. For a classification step whose output another system parses, a differently-shaped response is not degradation, it is a second failure with a success status code. Decide per feature whether a worse answer beats an honest error, and evaluate the fallback against the same test set as the primary rather than assuming it will be fine. Practical points worth building in from the start: keep the fallback with a different provider where you can, since a same-provider fallback shares the outage you are protecting against; make failover visible in your metrics and logs so you learn you were degraded rather than discovering it in a complaint; cap retries so a provider incident becomes a slow response rather than a cost spike; and rehearse the switch deliberately, because a fallback path exercised only during an outage is a path nobody has tested.