Context Window

The context window is the maximum amount of text, measured in tokens, that an LLM can hold "in view" at once across the system prompt, conversation history, retrieved documents, and the model's own response — think of it as the model's working memory for a single request. Once a conversation or document exceeds the context window, older content must be dropped, summarized, or the request fails outright. Context windows have grown dramatically: early GPT-3 offered 4K tokens (~3,000 words), while modern frontier models like Claude Sonnet 4.5 and Gemini 2.5 Pro offer 200K to 1M+ token windows (hundreds of thousands of words — entire codebases or books). This matters directly to SaaS builders architecting AI features: a large context window lets you stuff more retrieved documents, conversation history, or a full codebase into a single prompt without complex chunking or summarization — but it doesn't mean "unlimited and free." Every token in the context window is billed as input tokens on every single call (even if it's the same system prompt repeated 50 times in a conversation), and very long contexts can increase latency and sometimes degrade accuracy on information buried in the middle (the "lost in the middle" effect, where models attend more reliably to content near the start and end of context). A concrete example: a coding assistant with a 200K-token context window can be given an entire mid-sized codebase (say, 150K tokens of source files) plus a 500-token question like "where is the user authentication logic and does it handle token refresh?" — the model can answer directly by reading the whole codebase in one pass, something impossible with a 4K window that would require chunking and RAG instead. Builders manage context windows via techniques like sliding-window truncation (drop oldest messages), summarization (compress old turns into a shorter summary), and RAG (retrieve only the relevant slice instead of loading everything) — the right choice depends on whether completeness or cost/latency matters more for the use case. Context window size also interacts directly with pricing tiers: many providers charge more per token once a request crosses a certain context-length threshold (reflecting the genuinely higher compute cost of attending over more tokens), so a builder naively stuffing an entire 100-page document into every request when only one page is relevant pays both a latency and a cost penalty for content the model never needed to see. This is the core argument for RAG over "just use a bigger context window": even with a 1M-token window technically available, retrieving only the 3-5 most relevant chunks for a given question is almost always faster, cheaper, and often more accurate than dumping an entire corpus into context and hoping the model finds the right needle in an enormous haystack.