Fluxx Tech
Googlebot does not crawl every URL on your site every day. It allocates effort based on how fast your server responds and how much it values your content. That allocation is what people call crawl budget.
For a 40-page business website, this is irrelevant: Google will crawl everything. For a site with thousands of URLs, it decides whether your new product page gets seen this week or next month.
Who needs to care
You can ignore crawl budget if:
• Your site has a few hundred pages
• New pages get indexed within a few days
• Search Console shows no large “Discovered – currently not indexed” group
You should pay attention if:
• You have thousands of URLs: ecommerce, listings, large catalogues
• Pages take weeks to appear in search
• You generate URLs automatically through filters or parameters
• Your “Discovered – currently not indexed” count is large and growing
• Your server is slow
The two halves
Crawl capacity. How much Googlebot can crawl without overloading your server. A fast server raises it; timeouts and 5xx errors lower it immediately.
Crawl demand. How much Google wants to crawl. Driven by how popular and how fresh your pages are. A page nobody links to and that never changes gets visited rarely.
You influence capacity with hosting and performance, and demand with content quality, freshness and links.
Read your crawl stats
In Search Console: Settings → Crawl stats. Look at:
• Total crawl requests over time
• Average response time. Rising response time usually means falling crawl rate
• By response: how many 200s, 301s, 404s, 5xx. A large share of non-200 responses is wasted budget
• By file type: if most crawl requests are for images, CSS and JavaScript, your HTML pages are getting a smaller share
• By purpose: discovery versus refresh
A healthy pattern: mostly 200 responses, stable or falling response times, and a reasonable share of discovery crawling if you publish regularly.
What wastes crawl budget
Waste | Example |
Parameter URLs | ?sort=price&colour=red&page=3, thousands of combinations |
Faceted navigation | Every filter combination as a crawlable URL. See our guide |
Session IDs in URLs | A new URL for every visitor |
Infinite calendars | A booking calendar generating a URL per date, forever |
Internal redirect chains | Every hop is a separate request |
Soft 404s | Google keeps rechecking them |
Duplicate pages | Multiple URLs for the same content |
Broken internal links | Crawled, found missing, repeat |
Huge uncompressed images | Slow responses reduce crawl rate |
Orphan pages | Crawled rarely because nothing links to them |
Fixes, highest return first
1. Speed up server response. This raises crawl capacity directly. Caching, a faster host and a CDN all help. Aim for consistently fast time-to-first-byte.
2. Stop generating junk URLs. Decide which parameter and filter combinations should exist as crawlable URLs. Usually: very few.
3. Block what should not be crawled in robots.txt: internal search results, cart and checkout, filter combinations you do not want indexed, admin paths.
Remember the distinction: robots.txt stops crawling; noindex stops indexing. A page blocked in robots.txt cannot be read, so Google cannot see a noindex tag on it. To remove a page from the index, allow crawling and use noindex. To stop wasting crawl on pages that were never indexed, use robots.txt.
4. Clean your sitemap. Only canonical, indexable, 200-status URLs. No redirects, no 404s, no noindexed pages. Split large sitemaps by section so you can see which section has indexing problems.
5. Fix redirect chains. Point every internal link at the final URL.
6. Remove or consolidate thin pages. Fewer, better pages get crawled more often.
7. Improve internal linking to the pages you care about. Pages linked from your homepage and main navigation get crawled far more than pages buried five clicks deep.
Nofollow and crawl control
Adding nofollow to internal links is not a reliable way to control crawling; Google may still discover and crawl those URLs. Use proper URL design, robots.txt and canonical tags instead.
What does not help
• Submitting the same sitemap repeatedly
• Requesting indexing for hundreds of URLs manually
• Changing crawl rate settings, where they still exist
• Adding “crawl-delay” directives, which Google ignores
• Buying links to “increase authority” so Google crawls more
A monthly routine for large sites
1. Check crawl stats: requests, response time, response codes
2. Check the “Discovered – currently not indexed” count
3. Sample 20 recently crawled URLs from your server logs; are they pages you care about?
4. Check for new parameter patterns appearing
5. Verify the sitemap still contains only good URLs
Server log analysis is the most accurate view available. Your hosting provider can usually give you access logs; filtering them for Googlebot shows exactly what is being crawled and how often.
Frequently asked questions
How do I increase my crawl budget? Make the server faster and the site smaller in terms of junk URLs. Both raise effective crawl coverage.
Does crawl budget affect rankings? Not directly. It affects how quickly new and updated pages are discovered, which affects results indirectly.
My site has 200 pages. Should I worry? No. Spend that time on content and links instead.
