The crawlers that matter most in 2026: GPTBot and OAI-SearchBot feed ChatGPT's answers and search citations; ClaudeBot feeds Claude; PerplexityBot feeds Perplexity's source slots; Google-Extended governs Gemini grounding (while ordinary Googlebot feeds AI Overviews); Bingbot powers Copilot. Each honors robots.txt directives under its own user-agent.
Two failure modes block them: explicit robots.txt disallows (often inherited from copy-pasted templates) and CDN/WAF bot challenges that serve JavaScript interstitials — which most AI crawlers cannot solve, so they see an empty page. Auditing both is the first step of any AI visibility fix.