AI Crawler

An AI crawler is an automated client operated by an AI company that fetches pages from the open web. Three quite different jobs get filed under the one name: collecting text to train a future model, fetching pages at answer time so an assistant can ground what it says in something current, and fetching one page because a user pasted its link into a chat. Publishers who treat the three as a single thing usually end up making a decision they did not intend to make. The distinction is the whole point. Blocking training collection keeps your writing out of the next model's weights, and does nothing whatsoever about an assistant quoting you today from a live fetch. Blocking the retrieval fetchers does the opposite: your content stays eligible for training corpora already collected, while you disappear from the answers people are asking for right now. Those are close to opposite trade-offs, and a single blanket rule picks one of them by accident. Control works the way it always has on the web. Vendors publish the user-agent strings they crawl with, frequently splitting them by purpose so the two decisions can be made separately, and you allow or disallow those agents in robots.txt. That mechanism is advisory: it depends on the operator choosing to honour it, it does nothing about your text being republished somewhere you do not control, and enforcing anything requires blocking by user agent or address at the server or edge. Names and tokens change more often than anything else here, so read the vendor's own published documentation rather than a blog post summarising it, and re-read it whenever you revisit the file. A rule written against an agent string that has since been retired is indistinguishable, from the outside, from no rule at all. Underneath the configuration is a commercial question that is not technical. Is your content valuable because it is read on your site, or because it is cited wherever the reader happens to be? Documentation, marketing pages and glossaries usually sit firmly in the second camp and get blocked by accident, copied from a template written for a publisher whose business is the pageview.