AI crawlers are bots operated by AI companies — such as GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot — that fetch web content to train models or to power AI answers and citations.
The crawlers that matter most: OAI-SearchBot and ChatGPT-User fetch pages for ChatGPT answers and search citations, while GPTBot collects training data; Claude-SearchBot serves Claude's search and ClaudeBot is its training crawler; PerplexityBot and Perplexity-User serve Perplexity; Google-Extended controls use for Gemini training and grounding, while ordinary Googlebot feeds AI Overviews; Bingbot powers Copilot. Each honors robots.txt directives under its own user-agent.
Two failure modes block them: explicit robots.txt disallows (often inherited from copy-pasted templates) and CDN/WAF bot challenges that serve JavaScript interstitials — which most AI crawlers cannot solve, so they see an empty page. Auditing both is the first step of any AI visibility fix.
Search crawlers versus training crawlers
Most AI companies run separate bots for separate jobs, and blocking the wrong one is a common mistake. Search and user-fetch bots put you in answers; training bots collect material to build models.
OpenAI — OAI-SearchBot and ChatGPT-User fetch pages for ChatGPT search and user requests; GPTBot is the training crawler.
Anthropic — Claude-SearchBot is the search crawler; ClaudeBot is the training crawler.
Perplexity — PerplexityBot builds the index and Perplexity-User fetches pages on request.
Google — AI Overviews come from the normal Googlebot index; Google-Extended is a robots.txt token controlling use for Gemini training and grounding, and does not affect Search.
Others — Applebot-Extended, CCBot and Meta-ExternalAgent are mainly training-related.
How to see which AI bots visit your site
Your server or CDN access logs record the user-agent of every request. Search them for the names above to see which AI crawlers are active and which paths they fetch. Because anyone can fake a user-agent string, most vendors also publish IP ranges you can check requests against.
The two ways sites accidentally block them
Auditing both is the first step of any AI visibility fix. The free AI Visibility Checker reports each crawler family as open, partly blocked or blocked.
robots.txt disallows — often inherited from a copied template, or a blanket rule aimed at training bots that also catches search bots.
CDN or firewall bot challenges — a JavaScript interstitial that most AI crawlers cannot pass, so they see an empty page even though robots.txt allows them.
Frequently asked questions
How do I block AI crawlers?
Add a robots.txt group for each crawler's user-agent with Disallow: /. Block the training crawlers (GPTBot, ClaudeBot, Google-Extended) if you want to opt out of training, but leave the search crawlers allowed if you want to stay visible in AI answers.
Do AI crawlers respect robots.txt?
The major vendors say their documented crawlers do. robots.txt is voluntary, though, so it is a request rather than enforcement. If you need to actually block a bot, do it at the firewall.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is the training crawler. ChatGPT search uses OAI-SearchBot, so blocking GPTBot opts you out of training without removing you from answers.