In-Context Learning

In-context learning (ICL) is the emergent capability of large language models to adapt to a new task, format, or pattern purely from information presented within the prompt at inference time — instructions, examples, or demonstrations — without any gradient updates, fine-tuning, or retraining of the model's underlying weights. It is the theoretical foundation underneath both zero-shot prompting (learning from an instruction alone) and few-shot prompting (learning from worked examples), and it's considered one of the most surprising and important emergent properties of transformer-based LLMs trained at scale — smaller models show little to no in-context learning ability, while sufficiently large models trained on diverse data can pick up novel formats, even somewhat arbitrary or invented ones, from just a handful of examples in a single prompt. For SaaS builders, understanding ICL clarifies why prompt engineering works at all and why it's often preferable to fine-tuning for many production tasks: since the model can "learn" a task instantly from prompt content, teams can adapt general-purpose foundation models to company-specific tasks (custom categorization schemes, brand voice, unusual output formats) without the cost, latency, and MLOps overhead of training and hosting a custom fine-tuned model — the "training data" is just the examples you put in the prompt, resent on every call. The trade-off versus fine-tuning is that ICL "forgets" the pattern the instant a new conversation starts (nothing persists beyond that one prompt/context window), and it consumes context window and token budget on every request, whereas fine-tuning bakes the pattern into the model's weights permanently, at the cost of a training pipeline and reduced flexibility. In practice, most production SaaS AI features rely entirely on in-context learning via well-designed prompts and only reach for fine-tuning once they have significant volume, a stable task definition, and cost/latency pressure that justifies the investment. Concrete worked example: a legal-tech tool needs to classify clauses using the company's proprietary 15-category taxonomy that no public model was ever trained on. Instead of fine-tuning, the team includes 2 examples per category (30 examples total) directly in the prompt, showing clause text mapped to their exact category labels. The model, having never seen this taxonomy during training, correctly classifies new clauses by pattern-matching to the examples' structure — a working demonstration of in-context learning solving a genuinely novel task instantly, at the cost of a longer, more expensive prompt on every single call. As the team's volume grows, this is also the natural decision point for evaluating fine-tuning: once a task's definition has stabilized (the taxonomy stops changing) and call volume is high enough that the recurring token cost of 30 in-context examples on every request outweighs the one-time cost of a fine-tuning run, baking the pattern into model weights becomes the more economical long-term choice — in-context learning is usually the right starting point, and fine-tuning the right optimization once the task and its economics are proven out.