Crawl-Budget

Crawl budget refers to the number of pages a website bot crawls within a given timeframe. It is determined by two factors: how much load the server can handle and how important Google considers the website to be.

This is not an issue for most SME websites. It becomes relevant starting from a few thousand addresses, such as in online shops with many filter and sorting views or on portals with automatically generated pages.

Crawl budget is wasted primarily through redirect chains, endless parameter variants, and pages with no utility. It can be controlled via the robots.txt, via NoIndex and via a clean canonical tag. A short loading time also helps, because the bot can process more pages in the same amount of time.

We target websites specifically to search terms. Learn more at SEO support.

Interested in other topics? Here we have a few more suitable explanations.