Quick answer · Updated July 29, 2026

What Is Crawl Budget?

Direct Answer
Google defines a site's crawl budget as the set of URLs Googlebot can crawl without overloading the host and wants to crawl based on demand. The two components are the crawl capacity limit and crawl demand. Most small, normally updated sites do not need a dedicated crawl-budget project.

Should You Worry About It?

Google's current guide is intended mainly for sites with roughly one million or more pages that change moderately often, sites with roughly ten thousand or more pages that change daily, or sites with many URLs in the “Discovered – currently not indexed” category. Google calls these rough classification estimates, not exact thresholds.

If new pages are found promptly and Search Console does not show a large discovery backlog, keep your sitemap current and focus on useful, unique content rather than trying to force more crawling.

First Checks for a Large Site

  • Remove or consolidate duplicate and unimportant URL variants.
  • Return 404 or 410 for permanently removed content.
  • Keep sitemaps current and avoid long redirect chains.
  • Investigate server errors, rate limiting, and slow responses.
  • Use Crawl Stats and server logs to confirm where requests go.

Need the implementation detail?

Continue with the large-site crawl budget guide for URL-inventory controls, logs, sitemaps, status-code decisions, and a measurement workflow. This answer intentionally stays short.

Primary Source

Google Crawling Infrastructure: Optimize your crawl budget ↗

Frequently Asked Questions

Usually not. Google describes crawl-budget management as advanced guidance for very large or rapidly changing sites and sites with many URLs reported as Discovered – currently not indexed.
No. Crawling retrieves a URL; Google then evaluates, consolidates, and decides whether the page is suitable for its index.
Use Search Console's Crawl Stats report for host-level trends and server logs for the URLs and responses actually requested.