Free AI visibility checker →

The AI Crawler List: Every Bot, What It Feeds, and How to Control It

All 15 AI crawlers that matter in 2026, split by the distinction most robots.txt guides miss: retrieval crawlers that put you in AI answers, and training crawlers that only feed model training.

User-agentCompanyTypeWhat it feeds
OAI-SearchBotOpenAI RETRIEVAL ChatGPT Search results and citations
ChatGPT-UserOpenAI RETRIEVAL (USER FETCH) Live page fetches when a ChatGPT user asks about your site
GPTBotOpenAI TRAINING OpenAI model training corpora
Claude-SearchBotAnthropic RETRIEVAL Claude Search results and citations
Claude-UserAnthropic RETRIEVAL (USER FETCH) Live fetches during Claude conversations
ClaudeBotAnthropic TRAINING Anthropic model training
PerplexityBotPerplexity RETRIEVAL Perplexity answers and citations
Perplexity-UserPerplexity RETRIEVAL (USER FETCH) Live fetches for Perplexity users
GooglebotGoogle RETRIEVAL Google Search, AI Overviews and AI Mode
Google-ExtendedGoogle TRAINING / GROUNDING Gemini training and grounding
BingbotMicrosoft RETRIEVAL Bing results, feeds Copilot answers
AmazonbotAmazon RETRIEVAL Alexa answers and Rufus shopping assistant
Meta-ExternalAgentMeta TRAINING / RETRIEVAL Meta AI
BytespiderByteDance TRAINING TikTok and Doubao models
CCBotCommon Crawl TRAINING The open Common Crawl corpus many models train on

The distinction that decides your AI visibility

Blocking a retrieval crawler removes you from that engine's answers. If OAI-SearchBot cannot reach you, ChatGPT Search cannot cite you, full stop. Blocking a training crawler costs you nothing in visibility; it only keeps your content out of future model training, which is a licensing decision, not an SEO one.

The most common mistake we see: copying a robots.txt snippet that blocks both families, then wondering why the site never appears in AI answers.

robots.txt to be cited (allow retrieval, block training)

User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: /

Keep Googlebot and Bingbot unblocked unless you want out of ordinary search as well; both also feed AI answer surfaces.

robots.txt is only half the story

A crawler your robots.txt allows can still be blocked by a firewall or CDN bot rule that serves challenge pages to non-browser user agents. Your site looks fine to you and returns nothing to the crawler, and no file on your server will ever tell you. The only way to know is to request your pages as the crawler and compare what comes back, which is exactly what our free checker does.

Which of these 15 can actually reach your site?

Free check, about a minute: robots.txt rules and real fetch tests as five different AI crawlers, including the firewall blocks this page just warned you about.

Run the free AI visibility check