Crawl Budget

Crawl budget is the amount of crawling a search engine is willing to spend on a site — informally the combination of how fast it can fetch pages without straining the server and how much it currently wants that site's content. It is not a figure anyone publishes and not a setting you configure; it is an emergent constraint you only notice when it binds. For most sites it never binds, and a good deal of advice about it is misapplied. A few hundred pages that change occasionally will be crawled thoroughly without anyone thinking about it. The constraint becomes real on large sites, on sites that generate addresses combinatorially, and on slow origins where each fetch is expensive enough that the engine backs off on its own. When it does bind, the cause is almost always waste rather than volume. Faceted navigation multiplying filters and sort orders into millions of distinct URLs. Session identifiers appended to every link. Calendars with an infinite supply of next months. Soft 404s that return a status of 200 for pages that do not exist, so every wrong address is a real page as far as the crawler can tell. Long redirect chains that spend several fetches to arrive at one document. The fixes run in that order too. Best is not producing the URLs at all; next is disallowing the parameter space in robots.txt so the crawler never asks; next is returning honest status codes so a missing page reads as missing. Keeping the pages you care about shallow and well linked internally matters more than any of it, because a page nothing links to competes for attention from a standing start. Evidence comes from two places, neither of them a third-party tool. Your own server logs show exactly which paths were requested, by which agent, and what they received, which is the only way to see the waste directly. Each engine's own crawl statistics show the aggregate, and a rising request count against a flat number of useful pages is the signal to go looking.