XML Sitemap

An XML sitemap is a file listing the URLs on a site that you want crawled, optionally with a last-modified date for each. It does not make anything rank and it does not guarantee indexing; what it does is make discovery cheap, which matters most for pages that are new, buried several clicks deep, or reachable only through search and filter interfaces a crawler will not operate. The protocol sets hard limits: a single file holds up to 50,000 URLs or 50 MB uncompressed, whichever it hits first, and larger sites chain files together behind a sitemap index. Point to the file from robots.txt so any crawler finds it without being told, and submit it in each engine's webmaster tooling as well, because that is also where you find out which URLs were rejected and why. The rule that gets ignored is that a sitemap should list only URLs you would be happy to see indexed: canonical, returning 200, not blocked in robots.txt, not carrying noindex. A sitemap contradicting the rest of the site is worse than a missing one, because it is an explicit statement of intent that disagrees with your canonical tags, and the engine now has to decide which of your two signals to believe. The last-modified date is only useful if it is honest. A build pipeline that stamps every URL with today's date on every deploy makes the field meaningless, and an engine that learns your dates are noise stops reading them. Derive it from real content changes, or leave it out entirely — an absent field costs nothing and a lying one costs you the signal. On a multilingual site the sitemap is also the most maintainable place to declare language alternates, since it keeps the mapping in one generated file instead of in head tags on every page in every locale. Whatever you do, generate the sitemap from the same source as your routing, so a page that exists cannot be missing from the file and a file entry cannot outlive the page.