Crawl Budget
The number of pages Googlebot will crawl on your site within a given timeframe.
Full definition
Crawl Budget is the number of URLs Googlebot is prepared to crawl on a site within a given time period. It depends on domain authority and server response speed.
- •Crawl budget is critical for:
- •Large sites (10,000+ pages): new pages may wait a long time for indexation.
- •E-commerce sites with faceted navigation: filters generate millions of URLs.
- •Sites with frequently updated content.
- •How to optimize Crawl Budget:
- •Block utility URLs in robots.txt (sort parameters, filters, session parameters).
- •Use canonical tags to consolidate duplicates.
- •Speed up the server (TTFB < 200 ms).
- •Fix redirect chains.
- •Remove or merge thin-content pages (< 300 words).