Crawl budget
The set of URLs on a site that Google can and wants to crawl; Google's guide to it is for large sites whose pages change often, and others don't need to read it.
Google can’t crawl every address on the web, so it limits the time and resources it spends crawling any one site. That allocation “is commonly called a site’s crawl budget” (Google Crawling Infrastructure). Google defines it as “the set of URLs that Google can and wants to crawl,” and treats each hostname as a site of its own, so www.example.com and code.example.com have separate budgets.
Two things set it. The crawl capacity limit is how much Google can crawl without overwhelming your server: it goes up while your site responds consistently and its response times hold steady or improve, and down when it slows or returns server errors. Crawl demand is how much Google wants to crawl. For Googlebot, it depends on your site’s size, how often it’s updated, page quality and relevance, compared with other sites. Every site starts with the same conservative limit, which Google adjusts over time when there’s demand and the site stays healthy.
Does it concern your site?
Google’s guide to crawl budget is “intended primarily for” these sites:
- Large sites (1 million+ unique pages) whose content changes about once a week
- Sites with 10,000+ unique pages whose content changes daily
- Sites with a large share of their URLs listed in Search Console as “Discovered - currently not indexed”
Google calls these numbers a rough estimate, not exact thresholds. If your site doesn’t have a large number of pages that change rapidly, or your pages seem to be crawled the same day they’re published, Google says you don’t need to read the guide. For Google Search, keeping your sitemap up to date and checking the Page Indexing report regularly “is adequate.”
Google’s advice for the sites it concerns
To make crawling more efficient, Google advises you to consolidate duplicate content, block pages you never want crawled with robots.txt, return a 404 or 410 status for pages removed for good, keep your sitemaps up to date and avoid long redirect chains. Don’t use noindex for this, since Google still requests the page before dropping it.
Related terms
Tools for it
Sources
- https://developers.google.com/crawling/docs/crawl-budget (Oct 12, 2026)
Updated Oct 12, 2026.