Speculative Decoding

Speculative decoding is an inference optimization that speeds up token generation without changing the model's output. A small, fast "draft" model proposes several tokens ahead; the large target model then verifies them all in a single forward pass, accepting the ones it agrees with and correcting the first mismatch. Because verifying several tokens at once is much cheaper than generating them one by one, you get the exact same output the big model would have produced, but often two to three times faster. It is a lossless trick — the output distribution is provably identical to standard decoding. SaaS builders rarely implement this themselves, but it explains why some API endpoints and self-hosted serving stacks (vLLM, TensorRT-LLM) deliver dramatically lower latency at the same quality. When evaluating an inference provider or self-hosting, ask whether speculative decoding — or its cousins like Medusa and EAGLE — is enabled: on your workload it can meaningfully cut both latency and cost per token, with no quality tradeoff to weigh against it. The property that makes it unusually safe to adopt is that the speedup is provably free of quality cost. The target model verifies every proposed token against its own distribution, so the accepted sequence is statistically identical to what standard decoding would have produced — this is not a quality-for-speed trade like quantization or a smaller model tier, and there is nothing to A/B test on output. What varies instead is how much speedup you actually get. The gain, commonly in the range of a modest multiple, depends entirely on the acceptance rate: how often the small draft model's guesses match what the large model would have chosen. Predictable, formulaic text — structured extraction, boilerplate code, constrained JSON — draws high acceptance and the largest wins. Highly creative or genuinely unpredictable generation draws frequent rejections, and each rejection wastes the draft work, which is why the technique helps far less on open-ended writing and can, in pathological cases, barely help at all. The draft model must also be well matched to the target; a mismatched or too-weak drafter lowers acceptance and eats the benefit. For most teams this is a property of the platform rather than a decision they make, since it is implemented inside the serving stack, not exposed as a request parameter — but it is worth asking about when comparing hosted providers or configuring a self-hosted server, because unlike most latency optimizations it costs you nothing in output fidelity and therefore has no downside to weigh.