Positional Encoding

Positional encoding is how a transformer learns where a token sits in a sequence. Attention is order-blind by construction: it computes relationships between every pair of tokens without any notion of which came first, so a raw transformer would read a sentence as a bag of words. Positional encoding fixes that by adding position information to the token representations — historically as fixed sinusoidal patterns, later as learned position vectors, and in most current models as a rotation applied inside the attention computation itself so that what the model sees is the relative distance between two tokens rather than their absolute index. The practical consequences show up at the edges of the context window. A model trained mostly on short sequences has seen few examples of very large positional offsets, so quality often degrades before the advertised context limit is reached rather than exactly at it — one of the reasons a retrieval system that supplies a small, well-chosen set of passages still outperforms one that pastes everything in. Position handling is also why the ordering of your prompt matters: instructions, retrieved context and the user's question occupy different distances from the tokens being generated, and moving a critical instruction from the middle of a long prompt to the end is a real, measurable change rather than a stylistic one.