[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"glossary-robots-txt::en":3,"gloss-cluster-robots-txt::en":23,"gloss-next-robots-txt::en":9},{"slug":4,"category":5,"name":6,"definition":7,"meta_desc":8,"faq":9,"schema_markup":9,"related":10},"robots-txt","dev-tools","robots.txt","robots.txt is a plain-text file served from the root of a domain that tells automated clients which paths they may request. It is the oldest convention in web publishing, formalised much later as the Robots Exclusion Protocol in RFC 9309, and it has always been advisory: it works because well-behaved crawlers choose to read it and obey it, not because anything enforces it.\n\nThe rule most people have backwards is that Disallow governs crawling, not indexing. A URL you block can still be listed in results if other sites link to it, because the engine learns the address without ever fetching the content — which is how a blocked page ends up in an index with no description at all. To keep a page out of an index it has to be crawlable and carry a noindex directive, or sit behind authentication.\n\nThe structure is a set of groups keyed by User-agent, each with Disallow and Allow paths, plus one or more Sitemap lines that are read independently of any group. Per-agent groups are the mechanism for treating search crawlers and AI crawlers differently, and on the major engines the more specific matching rule wins rather than the first one written.\n\nThe self-inflicted wounds are consistent across teams. A staging Disallow: \u002F that ships to production and quietly removes a whole domain. Blocking a stylesheet or script directory so the crawler cannot render the page the way a person sees it. Treating the file as privacy: it is public, and a list of paths you would rather nobody visited is an excellent map for anyone curious.\n\nKeep it in version control with the rest of the site, review it whenever routing changes, and test it against each engine's own checker before and after a deploy. It is one small file with the power to take an entire site out of search, and the failure is silent for as long as it takes someone to notice the traffic.","robots.txt tells crawlers which paths they may request. It governs crawling, not indexing — and a blocked page can still be listed without a description.",null,[11,14,17,20],{"slug":12,"name":13},"cdn","Content Delivery Network (CDN)",{"slug":15,"name":16},"cms","Content Management System (CMS)",{"slug":18,"name":19},"rate-limiting","Rate Limiting",{"slug":21,"name":22},"static-site-generator","Static Site Generator (SSG)",[24,28,32,35,38,41,44,47,50,53,56,59],{"slug":25,"category":5,"name":26,"updated_at":27},"abstraction-layer","Abstraction Layer","2026-09-08T21:32:37+00:00",{"slug":29,"category":5,"name":30,"updated_at":31},"agent","Agent","2026-09-08T21:32:35+00:00",{"slug":33,"category":5,"name":34,"updated_at":31},"ai-code-assistant","AI Coding Assistant",{"slug":36,"category":5,"name":37,"updated_at":31},"api-gateway","API Gateway",{"slug":39,"category":5,"name":40,"updated_at":31},"api-versioning","API Versioning",{"slug":42,"category":5,"name":43,"updated_at":31},"autonomous-agent","Autonomous Agent",{"slug":45,"category":5,"name":46,"updated_at":31},"blue-green-deployment","Blue-Green Deployment",{"slug":48,"category":5,"name":49,"updated_at":31},"canary-deployment","Canary Deployment",{"slug":51,"category":5,"name":52,"updated_at":27},"canonical-url","Canonical URL",{"slug":54,"category":5,"name":55,"updated_at":31},"chaos-engineering","Chaos Engineering",{"slug":57,"category":5,"name":58,"updated_at":31},"ci-cd","Continuous Integration \u002F Continuous Deployment (CI\u002FCD)",{"slug":60,"category":5,"name":61,"updated_at":62},"circuit-breaker","Circuit Breaker","2026-09-08T21:32:36+00:00"]