Updated 31 October 2026
What Is Crawl Budget?
Crawl Capacity Limit vs. Crawl Demand
| Component | Question It Answers | Influenced By |
|---|---|---|
| Crawl capacity limit (crawl rate) | "How hard can Google crawl without overloading the site?" | Server response time, error rates, sustained responsiveness, Google's own crawling resources |
| Crawl demand | "How much does Google want to crawl?" | Perceived inventory, popularity, staleness, site events such as migrations |
Every URL Googlebot requests counts toward crawl budget: duplicate and parameter URLs, redirects, alternate versions such as hreflang variants, and resources like CSS, JavaScript and XHR requests that a page loads.
Crawling is not indexing. Crawling retrieves a URL; indexing is a later evaluation and consolidation step. An exclusion status should be investigated rather than assigned to crawl budget by default.
Crawl budget applies per hostname: www.example.com and shop.example.com are crawled separately.
When Crawl Budget Matters
Google's crawl-budget guide is intended mainly for these sites. Google calls the figures rough estimates, not exact thresholds:
- 1M+ unique pages with content that changes moderately often (weekly)
- 10K+ unique pages with rapidly changing content (daily)
- Large % of URLs classified as "Discovered – currently not indexed"
If new pages are found promptly and Search Console does not show a large discovery backlog, keep your sitemap current and focus on useful, unique content rather than trying to force more crawling. Page count alone does not prove a crawl-budget problem.
Evidence Worth Investigating
- A large share of URLs reported as "Discovered – currently not indexed". The status alone does not identify the cause.
- High crawl volume on low-value templates such as filters and sort orders while important pages are crawled rarely.
- Redirect chains inflating crawl requests; each hop is a separate request.
- Many soft 404s and thin or duplicate URLs being re-crawled.
How to Check Crawl Budget in Search Console
- Open Settings → Crawl stats for the property. It shows total crawl requests, total download size and average response time for the last 90 days.
- Break requests down by response, file type, purpose (discovery or refresh) and Googlebot type. A large share of redirects, 404s or 5xx responses shows where requests are wasted.
- Check the Host status panel for robots.txt fetch, DNS and server connectivity problems. Google slows crawling when the host is unhealthy.
- In the Page indexing report, a growing "Discovered – currently not indexed" count can mean Google found URLs but has not crawled them yet.
- For the URLs actually requested, use server or CDN logs filtered to verified Googlebot. Crawl Stats is a host-level sample, not a complete URL list.
Common Causes of Wasted Crawling
| Source | Example | Typical Control |
|---|---|---|
| Faceted navigation | /shoes?color=red&size=9&sort=price | Keep indexable filter combinations deliberate; block crawl-only parameter patterns that should never be fetched. |
| Duplicate URL variants | Tracking parameters, session IDs, trailing-slash and case variants | Link to one version internally; redirect or canonicalize the rest. |
| Infinite spaces | Calendar pages or endless pagination with no content | Stop linking to empty pages and return 404 where nothing exists. |
| Redirect chains | Old URL → interim URL → final URL | Point internal links and sitemaps at the final URL. |
| Soft 404s | A removed product that returns 200 with "not found" text | Return a real 404 or 410 for permanently removed content. |
Stores generate most of these through filters and variants. The ecommerce SEO audit checklist covers how to review them.
How to Optimize Crawl Budget
First sprint
- Build a URL inventory segmented by template and value, and identify the largest crawl sinks.
- Fix systemic redirect chains and internal links that point to redirected URLs.
- Resolve soft-404 patterns with real 404/410 responses or proper content.
- Choose faceted-navigation controls from each URL's purpose; consolidate duplicates and block only spaces that should not be crawled.
Engineering track
- Improve server stability and response times; reduce DNS, network and 5xx errors.
- Use crawlable
<a href>links for internal navigation.
Discovery
- Keep sitemaps focused on canonical, indexable URLs with accurate
<lastmod>values. - Give each paginated page a unique, self-canonical URL without fragments.
Noindex vs. Robots.txt vs. 404
| Method | Use When | Crawl Budget Effect |
|---|---|---|
| robots.txt | URLs that are never needed for search | Stops requests to those URLs. Not a way to temporarily move budget elsewhere. |
| noindex | Page must exist for users but should not appear in search | No savings: Google must crawl to see noindex |
| 404/410 | Content is permanently removed | Clear signal to stop re-crawling over time |
Measuring Crawl Budget Changes
Compare windows long enough to cover normal crawl variation, and record deployments, outages, migrations, and major publishing events.
- Crawl requests per day: overall and by template.
- Average response time and host status: confirm server improvements reach crawling.
- Discovery vs. refresh mix: read alongside releases, sitemap changes and new URL patterns.
- Crawl waste share: redirects, 4xx and soft 404s as a percentage of all requests.
- "Discovered – currently not indexed" trend: interpret with quality, duplication and technical signals.
- Time to first crawl: check new and updated pages with URL Inspection.
Primary Source
Google Crawling Infrastructure: Optimize your crawl budget ↗Related Guides
Frequently Asked Questions
Ready to Scale Your SEO?
Generate optimized content, review it with SEO checks, and publish to WordPress from one workflow.
Start 3-Day Free Trial