Pretraining

Pretraining is the first and by far the most expensive stage of building a language model: the base network is trained on a very large, broad corpus with a simple objective — predict the next token — until it has absorbed grammar, facts, code, and a great deal of latent reasoning ability. Everything that comes later (instruction tuning, preference optimisation, safety alignment) reshapes behaviour that pretraining already installed; none of it adds knowledge at anything like the same scale. That ordering explains several things builders run into. A model cannot reliably answer questions about material that was never in its pretraining data, which is why a knowledge cutoff exists and why retrieval is the standard fix rather than more fine-tuning. Fine-tuning changes style, format and task-following far more easily than it changes what the model knows, because a fine-tuning set is orders of magnitude smaller than a pretraining corpus. And the cost profile is lopsided: pretraining is a capital project measured in cluster-months, while adapting a pretrained model is something a small team can do. For almost every SaaS product the practical decision is not whether to pretrain — it is which pretrained base to build on, and how much of the remaining gap to close with prompting, retrieval, or a light adaptation pass. Pretraining from scratch is justified mainly when the domain vocabulary is genuinely unlike public text, or when the data cannot leave the building at all.