AI Crawlers (AI Bots)
AI crawlers are automated programs run by large AI providers that browse websites in order to collect training data or index content for AI search answers. Well-known examples are GPTBot from OpenAI, ClaudeBot from Anthropic and PerplexityBot. Website operators can use robots.txt to control which of these bots may crawl their pages, an important lever for being cited in AI search answers at all.
In practice
If you want to be visible in AEO and GEO (Answer Engine Optimization and Generative Engine Optimization), you should deliberately allow AI crawlers rather than blocking them across the board, bearing in mind that a block in robots.txt is only a request that no bot is technically obliged to follow. Many providers also distinguish between training bots such as GPTBot and retrieval bots for real-time answers such as ChatGPT-User, so the two can be allowed or blocked separately. Do not expect a uniform effect, though: OpenAI notes that ChatGPT-User covers user-initiated actions and that robots.txt rules may therefore not apply, pointing to OAI-SearchBot instead as the way to control automated crawling. Google-Extended, in turn, is not a separate user agent at all but only a robots.txt token that governs whether crawled content may be used for training and grounding Gemini models; Google states explicitly that it affects neither inclusion in Google Search nor ranking there. An additional llms.txt file is not yet an official web standard, but some tools already read it.