noindex

noindex is a directive that asks a search engine to keep a page out of its index. It is set either as a robots meta tag in the document head or as an X-Robots-Tag HTTP response header, which is the only route available for files that have no HTML head — PDFs, images, generated exports. Unlike a canonical, it is treated as an instruction rather than a hint. The trap worth internalising is that noindex requires the page to be crawled. If the same URL is disallowed in robots.txt, the crawler never fetches it, never sees the tag, and the page can stay listed indefinitely on the strength of inbound links alone. Blocking and de-indexing look like the same intention and are, in that combination, opposite instructions. The pages that usually deserve it are the ones a site generates structurally rather than writes: internal search result pages, empty tag and category listings, filter and sort permutations, thin paginated tails, staging and preview environments, and views that only make sense to a signed-in user. Each of these can be produced in the thousands by a template nobody thinks of as content. What it does not do is worth stating plainly. It is not access control: the page still serves in full to anyone who has the URL, and to any client that ignores the directive. It is not nofollow either — links on a noindexed page are still crawled and followed unless you say otherwise, which is often exactly what you want for a page that exists only to route. Removal is not instant in either direction. A page leaves the index when the engine next crawls it, so the tag must stay in place until then rather than being swapped for a robots.txt block. And because engines eventually crawl long-unchanged noindexed pages less often, taking the tag back off can also take a while to have any visible effect.