[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"glossary-latency::en":3,"gloss-cluster-latency::en":20,"gloss-next-latency::en":9},{"slug":4,"category":5,"name":6,"definition":7,"meta_desc":8,"faq":9,"schema_markup":9,"related":10},"latency","core-ai","Latency","Latency, in the context of AI systems, is the time elapsed between a request being sent to a model (an API call, or a local inference request) and the response being received — commonly broken down into time-to-first-token (TTFT, how long before any output starts appearing) and total generation time (how long until the full response completes). Latency is one of the most consequential, under-discussed constraints in production AI products, because it directly shapes what UX patterns are even viable: a feature with 200ms latency can feel instant and be triggered on every keystroke (like autocomplete), while a feature with 8-second latency needs a loading state, can't run synchronously in a critical user flow, and might need to be moved to an asynchronous\u002Fbackground job instead. Latency is driven by several compounding factors: model size (larger models take longer per token generated), output length (more tokens to generate means more time, since most LLMs generate one token at a time autoregressively), network round-trip time to the API provider, server load\u002Fqueueing at the provider, and whether streaming is used (streaming sends tokens as they're generated rather than waiting for the full response, dramatically improving perceived latency even when total generation time is unchanged). A concrete worked example: a code-editor AI autocomplete feature has a hard latency budget of roughly 200-300ms to feel responsive while typing — this rules out large frontier models for that specific feature (a Claude Opus-class call might take 1-3+ seconds) and pushes builders toward small, fast, often locally-run or edge-deployed models specifically optimized for this latency budget, even accepting somewhat lower output quality as the trade-off. Contrast that with a \"generate my quarterly report\" feature, where users expect to wait and a 10-15 second response with a progress indicator is perfectly acceptable, allowing the builder to use a much larger, higher-quality model. Builders manage latency through model selection (smaller\u002Fdistilled models for latency-critical paths), streaming responses to improve perceived speed, prompt caching (skipping recomputation for repeated context), and architectural choices like running latency-tolerant tasks (batch summarization, report generation) asynchronously rather than blocking the UI. Latency also compounds across a multi-step AI pipeline in ways that are easy to underestimate: a RAG feature involving an embedding call, a vector-database query, a reranking call, and finally the LLM generation call has four sequential latency contributors, not one — and a naive implementation that runs these steps strictly in sequence can produce a noticeably sluggish user experience even if each individual step is reasonably fast. Production RAG and agentic systems commonly parallelize independent steps (e.g., running multiple retrieval queries concurrently) and use streaming aggressively to make the cumulative latency feel acceptable, since perceived latency (when the user first sees something happening) matters as much to UX as total completion time.","Latency is the delay between sending a request to an AI model and receiving a response — a critical constraint for any real-time AI feature.",null,[11,14,17],{"slug":12,"name":13},"inference","Inference",{"slug":15,"name":16},"quantization","Quantization",{"slug":18,"name":19},"throughput","Throughput",[21,25,29,33,36,39,42,45,48,51,54,57],{"slug":22,"category":5,"name":23,"updated_at":24},"agentic","Agentic AI","2026-08-24T02:46:36+00:00",{"slug":26,"category":5,"name":27,"updated_at":28},"alignment-tax","Alignment Tax","2026-08-24T02:46:37+00:00",{"slug":30,"category":5,"name":31,"updated_at":32},"artificial-intelligence","Artificial Intelligence (AI)","2026-08-24T02:46:38+00:00",{"slug":34,"category":5,"name":35,"updated_at":24},"attention","Attention",{"slug":37,"category":5,"name":38,"updated_at":32},"beam-search","Beam Search",{"slug":40,"category":5,"name":41,"updated_at":28},"benchmark-contamination","Benchmark Contamination",{"slug":43,"category":5,"name":44,"updated_at":28},"catastrophic-forgetting","Catastrophic Forgetting",{"slug":46,"category":5,"name":47,"updated_at":32},"computer-vision","Computer Vision",{"slug":49,"category":5,"name":50,"updated_at":28},"constitutional-ai","Constitutional AI",{"slug":52,"category":5,"name":53,"updated_at":24},"context-window","Context Window",{"slug":55,"category":5,"name":56,"updated_at":32},"deep-learning","Deep Learning",{"slug":58,"category":5,"name":59,"updated_at":24},"diffusion-model","Diffusion Model"]