Temperature

Temperature is a parameter passed at inference time that controls how "random" or "deterministic" an LLM's token selection is. Internally, the model computes a probability distribution over every possible next token; temperature rescales that distribution before a token is sampled. A temperature of 0 (or very close to it) makes the model almost always pick the single highest-probability token — deterministic, focused, repeatable output. Higher temperatures (1.0 and above) flatten the probability distribution, giving lower-probability tokens a real chance of being selected, producing more varied, creative, sometimes surprising output — but also a higher risk of incoherence or hallucination at the extremes. This is one of the most practically important tuning knobs for SaaS builders because the right temperature depends entirely on the task. Structured, factual, or code-generation tasks (SQL generation, data extraction, classification, API-response formatting) want low temperature (0 to 0.3) for consistency and correctness — you don't want your JSON-schema extractor to occasionally get creative with field names. Creative tasks (marketing copy brainstorming, name generation, story writing) benefit from higher temperature (0.7 to 1.0+) to avoid repetitive, generic output. A concrete example: asking an LLM "Give me a tagline for a coffee subscription box" at temperature 0 might reliably return "Freshly Roasted, Delivered to Your Door" every single time — safe but repetitive across many users. The same prompt at temperature 0.9 might return "Wake Up to Wonder" one call and "Your Daily Ritual, Reinvented" the next — more variety, useful when generating multiple options or avoiding a generic feel across thousands of users. Builders should be aware that temperature is not a magic "creativity" dial that improves quality — it only controls sampling randomness, so pushing it too high on tasks requiring precision (extracting a phone number from text, generating valid code) increases error rates without any upside. Temperature is often combined with top-p sampling for finer control, and most APIs recommend adjusting one or the other, not both simultaneously. Some providers also expose related parameters worth knowing alongside temperature: `frequency_penalty` and `presence_penalty` (available on some APIs) discourage the model from repeating the same words or topics, useful for long-form generation that otherwise gets repetitive; and a `seed` parameter (where supported) can improve — though rarely guarantee — reproducibility across calls at a fixed temperature, useful for testing and debugging. Builders running automated evaluation pipelines on model output typically fix temperature to 0 specifically so that test runs are comparable across code changes, only relaxing it once a feature has moved from evaluation into live user-facing generation where some variety is actually desirable.