Crawl Budget

Crawl budget is the amount of crawling a search engine can and wants to spend on a site. Crawl focus is the practical SEO question: are crawlers spending time on important, canonical, indexable pages?

For small sites, crawl budget is rarely the first problem. For large ecommerce, marketplace, publisher, international, or faceted-navigation sites, duplicate URL spaces and errors can waste crawl attention.

Common crawl waste

• Facet and filter combinations.
• Sorting and tracking parameters.
• Internal search pages.
• Redirect chains.
• Broken internal links.
• Non-canonical URLs in sitemaps.
• Duplicate hostnames or protocol variants.
• Large archives with little standalone value.

Improving crawl focus usually requires internal-link cleanup, sitemap cleanup, canonical alignment, redirects, robots controls, noindex decisions, and better page quality.