Natural Language Processing (NLP)

Natural language processing is the field concerned with getting computers to work with human language — classifying it, extracting structure from it, translating it, summarising it, generating it. It predates the current model era by decades, and the history matters because it explains what the tooling looks like today. The first generation was rule-based: grammars, parsers, hand-curated lexicons, and a great deal of linguistic expertise encoded by hand. The second was statistical, treating language as a distribution to be estimated from corpora. The third is neural, and large language models have absorbed so many previously separate tasks that "NLP pipeline" now often means one model call. That absorption is the practical headline. Named-entity recognition, sentiment classification, translation, summarisation, and question answering each used to be its own model with its own training set and its own evaluation. A capable general model now does all of them acceptably from a prompt, which collapses months of task-specific work into an afternoon. But acceptably is not always sufficiently, and this is where teams get burned: a purpose-built classifier trained on your own labelled data will usually beat a general model on your specific distribution, run for a fraction of the cost, return in milliseconds, and behave the same way tomorrow as it did today. The classical vocabulary is still worth knowing because it names the failure modes you will hit. Tokenisation decides what the model even sees, and it is why character counts and word counts diverge from billing. Ambiguity — lexical, syntactic, referential — is why a summary can be fluent and wrong. Domain shift is why a model that handles product reviews stumbles on clinical notes. And evaluation is where NLP is genuinely hard: there is no single correct summary, so judging generated language means either human review, a scoring rubric, or a model judging a model, each with its own bias. Choosing between a general model and a trained one is mostly a question of volume, latency budget, and how tightly you need behaviour pinned.