Crawl Budget

Crawl budget is the number of URLs that Googlebot can and wants to crawl on a website within a given period. It results from the crawl capacity limit (how much time and how many parallel connections the server can offer Googlebot without becoming overloaded) and crawl demand (how relevant and up to date Google considers the content).

In practice

For most small and medium-sized websites, crawl budget is not a problem, because Google crawls them fully anyway. According to Google it becomes relevant above all on very large sites with frequently changing content, or on sites with many technically unnecessary URLs, for instance from filter combinations or duplicate content. Wasting crawl budget risks new or updated pages being indexed late or not at all. Countermeasures: block unnecessary URLs via robots.txt or consolidate them with a canonical, improve server response times and avoid redirect chains.

Matching service

Sources

← Back to the glossary