Crawling & Indexing

Crawling is the automated process in which search engine bots such as Googlebot visit web pages and follow links; indexing is the subsequent addition of the content found to the searchable database of a search engine – only indexed pages can appear in search results.

In practice

Clear internal linking, a current XML sitemap and a correctly configured robots.txt guide crawlers to the important pages and preserve the crawl budget of large websites. Whether a crawled page is then indexed as well depends on content quality, canonical signals and the meta robots tag; you can check both crawling and indexing status for individual URLs in Google Search Console.

Matching service

Sources

← Back to the glossary