Skip to content

Free tool

Every major AI crawler, explained

Who runs each bot, what it collects, its robots.txt token, whether it follows robots.txt, and how to tell the real one from an impostor.

50 crawlers. Each one checked against its operator's documentation; last reviewed 4 November 2026.

Showing 50 of 50 crawlers

Build a robots.txt from this list

AI training

Collects pages to train AI models. Blocking it does not remove you from AI answers.

  • GPTBot

    GPTBot

    OpenAI

    Follows robots.txtIP list

    GPTBot is OpenAI's crawler that collects public pages to train its foundation models.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
    Robots.txt
    Yes, according to OpenAI.
    How to verify
    Check the IP against OpenAI's published list. OpenAI's IP list
    Block GPTBot
    User-agent: GPTBot
Disallow: /
    Allow GPTBot
    User-agent: GPTBot
Allow: /

    About GPTBotVerify GPTBot requests

  • ClaudeBot

    ClaudeBot

    Anthropic

    Follows robots.txtIP list

    ClaudeBot is Anthropic's crawler that collects public pages to train its Claude models.

    Sample user agent
    Anthropic does not publish its user agent strings. Trackers report strings containing ClaudeBot; match on the token, not a full string.
    Robots.txt
    Yes, according to Anthropic. Anthropic says it also aims to respect Crawl-delay.
    How to verify
    Check the IP against Anthropic's published list. Anthropic publishes one list shared by all its bots. Anthropic's IP list
    Block ClaudeBot
    User-agent: ClaudeBot
Disallow: /
    Allow ClaudeBot
    User-agent: ClaudeBot
Allow: /

    About ClaudeBotVerify ClaudeBot requests

  • anthropic-ai

    anthropic-ai

    Anthropic

    Not documentedNo way to verifyOlder token

    anthropic-ai is an older Anthropic robots.txt token that still appears in many templates.

    Sample user agent
    Anthropic's current crawler page does not mention this token. Trackers call it a legacy training token.
    Robots.txt
    Not documented by the operator. Anthropic's current page does not mention it.
    How to verify
    Anthropic doesn't publish a way to verify it.
    Purpose
    Anthropic no longer documents this token. Trackers list it as a legacy AI training token.
    Block anthropic-ai
    User-agent: anthropic-ai
Disallow: /
    Allow anthropic-ai
    User-agent: anthropic-ai
Allow: /

    About anthropic-ai

  • Google-Extended

    Google-Extended

    Google

    Follows robots.txtToken only

    Google-Extended is a robots.txt token Google reads to decide whether your pages may be used for Gemini training and grounding. Google-Extended also controls Gemini grounding, so blocking it can keep your pages out of some Gemini answers. It does not affect Google Search.

    Sample user agent
    None. No crawler sends this user agent. Google reads the token from robots.txt.
    Robots.txt
    Yes, according to Google. It is a robots.txt control, not a crawler. Blocking it does not affect Google Search.
    How to verify
    Nothing to verify: no crawler sends this user agent.
    Block Google-Extended
    User-agent: Google-Extended
Disallow: /
    Allow Google-Extended
    User-agent: Google-Extended
Allow: /

    About Google-Extended

  • Applebot-Extended

    Applebot-Extended

    Apple

    Follows robots.txtToken only

    Applebot-Extended is a robots.txt token Apple reads to decide whether pages Applebot crawled may train its generative models.

    Sample user agent
    None. No crawler sends this user agent. Apple reads the token from robots.txt.
    Robots.txt
    Yes, according to Apple. Apple says it does not crawl webpages. Blocking it does not affect Apple search features.
    How to verify
    Nothing to verify: no crawler sends this user agent.
    Block Applebot-Extended
    User-agent: Applebot-Extended
Disallow: /
    Allow Applebot-Extended
    User-agent: Applebot-Extended
Allow: /

    About Applebot-Extended

  • CCBot

    CCBot

    Common Crawl

    Follows robots.txtIP list

    CCBot is Common Crawl's crawler that builds an open web dataset widely used to train AI models.

    Sample user agent
    CCBot/2.0 (https://commoncrawl.org/faq/)
    Robots.txt
    Yes, according to Common Crawl.
    How to verify
    Check the IP against Common Crawl's published list. Reverse DNS on crawl.commoncrawl.org, confirmed with a forward lookup. Reverse DNS works for IPv4 addresses only. Common Crawl's IP list
    Block CCBot
    User-agent: CCBot
Disallow: /
    Allow CCBot
    User-agent: CCBot
Allow: /

    About CCBotVerify CCBot requests

  • Meta-ExternalAgent

    Meta-ExternalAgent

    meta-externalagent

    Follows robots.txtNo way to verify

    Meta-ExternalAgent is Meta's crawler for training its AI models and indexing content.

    Sample user agent
    meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
    Robots.txt
    Yes, according to Meta.
    How to verify
    Meta doesn't publish a way to verify it.
    Block Meta-ExternalAgent
    User-agent: meta-externalagent
Disallow: /
    Allow Meta-ExternalAgent
    User-agent: meta-externalagent
Allow: /

    About Meta-ExternalAgent

  • FacebookBot

    FacebookBot

    Meta

    Not documentedNo way to verifyOlder token

    FacebookBot is an older Meta crawler token that Meta no longer lists on its crawler page.

    Sample user agent
    Meta's current crawler page does not list FacebookBot.
    Robots.txt
    Not documented by the operator. Meta's current crawler page does not list it.
    How to verify
    Meta doesn't publish a way to verify it.
    Purpose
    Meta no longer documents FacebookBot. Other sources disagree on its purpose (speech recognition training, search or previews); Referscope groups it with AI training, as the product does.
    Block FacebookBot
    User-agent: FacebookBot
Disallow: /
    Allow FacebookBot
    User-agent: FacebookBot
Allow: /

    About FacebookBot

  • Bytespider

    Bytespider

    ByteDance

    Not documentedNo way to verify

    Bytespider is a crawler that identifies itself as ByteDance and is widely reported to collect pages for AI training.

    Sample user agent
    Not documented by the operator. Trackers report a user agent like "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36"; treat it as unconfirmed.
    Robots.txt
    Not documented by the operator. ByteDance publishes no crawler documentation, and reports on whether it follows robots.txt disagree.
    How to verify
    ByteDance doesn't publish a way to verify it.
    Purpose
    ByteDance does not document it. The purpose comes from third-party trackers.
    Block Bytespider
    User-agent: Bytespider
Disallow: /
    Allow Bytespider
    User-agent: Bytespider
Allow: /

    About Bytespider

  • Amazonbot

    Amazonbot

    Amazon

    Follows robots.txtIP list

    Amazonbot is Amazon's crawler that improves its products and, Amazon says, may be used to train Amazon AI models.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
    Robots.txt
    Yes, according to Amazon. Amazon says it does not support Crawl-delay and caches robots.txt for up to 30 days.
    How to verify
    Check the IP against Amazon's published list. Amazon publishes its list as a web page of single IP addresses. Amazon's IP list
    Block Amazonbot
    User-agent: Amazonbot
Disallow: /
    Allow Amazonbot
    User-agent: Amazonbot
Allow: /

    About Amazonbot

  • MistralAI-Training

    MistralAI-Training

    Mistral

    Follows robots.txtNo way to verify

    MistralAI-Training is Mistral's crawler that collects public pages to train its models.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)
    Robots.txt
    Yes, according to Mistral.
    How to verify
    Mistral doesn't publish a way to verify it.
    Block MistralAI-Training
    User-agent: MistralAI-Training
Disallow: /
    Allow MistralAI-Training
    User-agent: MistralAI-Training
Allow: /

    About MistralAI-Training

  • ElevenlabsCrawler

    ElevenlabsCrawler

    ElevenLabs

    Follows robots.txtNo way to verify

    ElevenlabsCrawler is ElevenLabs' crawler that collects public pages to train its AI models.

    Sample user agent
    Mozilla/5.0 (compatible; ElevenlabsCrawler/1.0; +https://elevenlabs.io/docs/bots)
    Robots.txt
    Yes, according to ElevenLabs. ElevenLabs says changes take about 24 hours to apply.
    How to verify
    ElevenLabs doesn't publish a way to verify it.
    Block ElevenlabsCrawler
    User-agent: ElevenlabsCrawler
Disallow: /
    Allow ElevenlabsCrawler
    User-agent: ElevenlabsCrawler
Allow: /

    About ElevenlabsCrawler

  • cohere-ai

    cohere-ai

    Cohere

    Not documentedNo way to verify

    cohere-ai is a token attributed to Cohere that trackers associate with AI training or chat fetches.

    Sample user agent
    Not documented by the operator. Cohere publishes no crawler page; match on the token only.
    Robots.txt
    Not documented by the operator. Cohere publishes no crawler documentation.
    How to verify
    Cohere doesn't publish a way to verify it.
    Purpose
    Cohere does not document it. Trackers disagree on whether it collects training data or fetches pages for chat.
    Block cohere-ai
    User-agent: cohere-ai
Disallow: /
    Allow cohere-ai
    User-agent: cohere-ai
Allow: /

    About cohere-ai

  • cohere-training-data-crawler

    cohere-training-data-crawler

    Cohere

    Not documentedNo way to verify

    cohere-training-data-crawler is a token attributed to Cohere that trackers describe as a training data crawler.

    Sample user agent
    Not documented by the operator. Cohere publishes no crawler page; match on the token only.
    Robots.txt
    Not documented by the operator. Cohere publishes no crawler documentation.
    How to verify
    Cohere doesn't publish a way to verify it.
    Purpose
    Cohere does not document it. The purpose comes from third-party trackers.
    Block cohere-training-data-crawler
    User-agent: cohere-training-data-crawler
Disallow: /
    Allow cohere-training-data-crawler
    User-agent: cohere-training-data-crawler
Allow: /

    About cohere-training-data-crawler

  • Timpibot

    Timpibot

    Timpi

    Not documentedNo way to verify

    Timpibot is a crawler attributed to Timpi that trackers associate with AI training and a decentralised search index.

    Sample user agent
    Not documented by the operator. Trackers report different strings (Timpibot/0.9 and Timpibot/1.0); match on the token only.
    Robots.txt
    Not documented by the operator. Timpi publishes no crawler documentation, and reports disagree.
    How to verify
    Timpi doesn't publish a way to verify it.
    Purpose
    Timpi does not document it. The purpose comes from third-party trackers, which disagree.
    Block Timpibot
    User-agent: Timpibot
Disallow: /
    Allow Timpibot
    User-agent: Timpibot
Allow: /

    About Timpibot

AI search and answers

Indexes pages so AI answers can cite and link to you. Blocking it can remove you from those answers.

  • OAI-SearchBot

    OAI-SearchBot

    OpenAI

    Follows robots.txtIP list

    OAI-SearchBot is OpenAI's crawler that finds pages to show and link in ChatGPT search results.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
    Robots.txt
    Yes, according to OpenAI. OpenAI says robots.txt changes take about 24 hours to apply.
    How to verify
    Check the IP against OpenAI's published list. OpenAI's IP list
    Block OAI-SearchBot
    User-agent: OAI-SearchBot
Disallow: /
    Allow OAI-SearchBot
    User-agent: OAI-SearchBot
Allow: /

    About OAI-SearchBotVerify OAI-SearchBot requests

  • Claude-SearchBot

    Claude-SearchBot

    Anthropic

    Follows robots.txtIP list

    Claude-SearchBot is Anthropic's crawler that indexes pages to improve the quality of Claude's search results.

    Sample user agent
    Anthropic does not publish its user agent strings. Match on the token, not a full string.
    Robots.txt
    Yes, according to Anthropic.
    How to verify
    Check the IP against Anthropic's published list. Anthropic publishes one list shared by all its bots. Anthropic's IP list
    Block Claude-SearchBot
    User-agent: Claude-SearchBot
Disallow: /
    Allow Claude-SearchBot
    User-agent: Claude-SearchBot
Allow: /

    About Claude-SearchBotVerify Claude-SearchBot requests

  • PerplexityBot

    PerplexityBot

    Perplexity

    Not yet verified by usIP list

    PerplexityBot is Perplexity's crawler that indexes pages to show and link in Perplexity's answers.

    Sample user agent
    Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
    Robots.txt
    Not yet verified by us. Perplexity explains how to block it and says changes take up to 24 hours; we have not yet confirmed a plain statement that it follows robots.txt.
    How to verify
    Check the IP against Perplexity's published list. Perplexity's IP list
    Block PerplexityBot
    User-agent: PerplexityBot
Disallow: /
    Allow PerplexityBot
    User-agent: PerplexityBot
Allow: /

    About PerplexityBotVerify PerplexityBot requests

  • DuckAssistBot

    DuckAssistBot

    DuckDuckGo

    Follows robots.txtIP list

    DuckAssistBot is DuckDuckGo's crawler that fetches pages for its AI-assisted answers; DuckDuckGo says it is not used to train AI models.

    Sample user agent
    DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
    Robots.txt
    Yes, according to DuckDuckGo. DuckDuckGo says opting out takes up to 72 hours.
    How to verify
    Check the IP against DuckDuckGo's published list. Its list matches DuckDuckBot's, so the IP alone cannot tell the two apart. DuckDuckGo's IP list
    Block DuckAssistBot
    User-agent: DuckAssistBot
Disallow: /
    Allow DuckAssistBot
    User-agent: DuckAssistBot
Allow: /

    About DuckAssistBotVerify DuckAssistBot requests

  • Meta-WebIndexer

    Meta-WebIndexer

    meta-webindexer

    Follows robots.txtNo way to verify

    Meta-WebIndexer is Meta's crawler that indexes pages so Meta AI can cite them in its answers.

    Sample user agent
    meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
    Robots.txt
    Yes, according to Meta.
    How to verify
    Meta doesn't publish a way to verify it.
    Block Meta-WebIndexer
    User-agent: meta-webindexer
Disallow: /
    Allow Meta-WebIndexer
    User-agent: meta-webindexer
Allow: /

    About Meta-WebIndexer

  • Amzn-SearchBot

    Amzn-SearchBot

    Amazon

    Follows robots.txtIP list

    Amzn-SearchBot is Amazon's crawler for Alexa and Amazon search experiences; Amazon says it is not used for training.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
    Robots.txt
    Yes, according to Amazon. Amazon says it falls back to your rules for other search crawlers when it has no group of its own.
    How to verify
    Check the IP against Amazon's published list. Amazon publishes its list as a web page of single IP addresses. Amazon's IP list
    Block Amzn-SearchBot
    User-agent: Amzn-SearchBot
Disallow: /
    Allow Amzn-SearchBot
    User-agent: Amzn-SearchBot
Allow: /

    About Amzn-SearchBot

  • MistralAI-Index

    MistralAI-Index

    Mistral

    Follows robots.txtIP list

    MistralAI-Index is Mistral's crawler that builds its search index; Mistral says it is not used for training.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
    Robots.txt
    Yes, according to Mistral.
    How to verify
    Check the IP against Mistral's published list. Mistral's IP list
    Block MistralAI-Index
    User-agent: MistralAI-Index
Disallow: /
    Allow MistralAI-Index
    User-agent: MistralAI-Index
Allow: /

    About MistralAI-IndexVerify MistralAI-Index requests

  • YouBot

    YouBot

    You.com

    Follows robots.txtSigned requests

    YouBot is You.com's crawler that powers You.com search and its AI answers.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36
    Robots.txt
    Yes, according to You.com. You.com says it respects Crawl-delay and caches robots.txt for 30 minutes.
    How to verify
    You.com signs its requests (checking a signature needs the request headers, not a log line). Reverse DNS on search.you.com, confirmed with a forward lookup. Check the IP against You.com's published list. You.com publishes a single range, 68.67.112.0/24, on its documentation page. You.com's IP list
    Block YouBot
    User-agent: YouBot
Disallow: /
    Allow YouBot
    User-agent: YouBot
Allow: /

    About YouBotVerify YouBot requests

User-triggered fetch

Fetches a page when a person asks an assistant about it.

  • ChatGPT-User

    ChatGPT-User

    OpenAI

    Partly followsIP list

    ChatGPT-User is the agent OpenAI uses when a ChatGPT user or a custom GPT asks to read a page.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
    Robots.txt
    Partly. OpenAI says robots.txt rules may not apply, because a person asked for the page.
    How to verify
    Check the IP against OpenAI's published list. OpenAI's IP list
    Block ChatGPT-User
    User-agent: ChatGPT-User
Disallow: /
    Allow ChatGPT-User
    User-agent: ChatGPT-User
Allow: /

    About ChatGPT-UserVerify ChatGPT-User requests

  • Claude-User

    Claude-User

    Anthropic

    Follows robots.txtIP list

    Claude-User is the agent Anthropic uses when a Claude user asks a question that needs a page read.

    Sample user agent
    Anthropic does not publish its user agent strings. Match on the token, not a full string.
    Robots.txt
    Yes, according to Anthropic.
    How to verify
    Check the IP against Anthropic's published list. Anthropic publishes one list shared by all its bots. Anthropic's IP list
    Block Claude-User
    User-agent: Claude-User
Disallow: /
    Allow Claude-User
    User-agent: Claude-User
Allow: /

    About Claude-UserVerify Claude-User requests

  • Perplexity-User

    Perplexity-User

    Perplexity

    Ignores robots.txtIP list

    Perplexity-User is the agent Perplexity uses when a user asks it to read a page.

    Sample user agent
    Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
    Robots.txt
    No. Perplexity says it generally ignores robots.txt rules, because a person asked for the page.
    How to verify
    Check the IP against Perplexity's published list. Perplexity's IP list
    Block Perplexity-User
    User-agent: Perplexity-User
Disallow: /
    Allow Perplexity-User
    User-agent: Perplexity-User
Allow: /

    About Perplexity-UserVerify Perplexity-User requests

  • Meta-ExternalFetcher

    Meta-ExternalFetcher

    meta-externalfetcher

    Partly followsNo way to verify

    Meta-ExternalFetcher is the agent Meta uses to fetch a link a person asked its AI to read.

    Sample user agent
    meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
    Robots.txt
    Partly. Meta says it may bypass robots.txt rules, because a person asked for the page.
    How to verify
    Meta doesn't publish a way to verify it.
    Block Meta-ExternalFetcher
    User-agent: meta-externalfetcher
Disallow: /
    Allow Meta-ExternalFetcher
    User-agent: meta-externalfetcher
Allow: /

    About Meta-ExternalFetcher

  • MistralAI-User

    MistralAI-User

    Mistral

    Follows robots.txtIP list

    MistralAI-User is the agent Mistral uses when a user asks its assistant to read a page; Mistral says it is not used for training.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
    Robots.txt
    Yes, according to Mistral. Mistral says the token governs which sites these user requests can be made to.
    How to verify
    Check the IP against Mistral's published list. Mistral's IP list
    Block MistralAI-User
    User-agent: MistralAI-User
Disallow: /
    Allow MistralAI-User
    User-agent: MistralAI-User
Allow: /

    About MistralAI-UserVerify MistralAI-User requests

  • Amzn-User

    Amzn-User

    Amazon

    Partly followsIP list

    Amzn-User is the agent Amazon uses to fetch a page live when a user asks for it.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36
    Robots.txt
    Partly. Amazon says it may not follow all robots.txt directives.
    How to verify
    Check the IP against Amazon's published list. Amazon publishes its list as a web page of single IP addresses. Amazon's IP list
    Block Amzn-User
    User-agent: Amzn-User
Disallow: /
    Allow Amzn-User
    User-agent: Amzn-User
Allow: /

    About Amzn-User

  • Google-Agent

    Google-Agent

    Google

    Ignores robots.txtIP list

    Google-Agent is the agent Google uses when a person asks a Google AI agent to act on a page.

    Sample user agent
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36
    Robots.txt
    No. Google says user-triggered fetchers generally ignore robots.txt rules.
    How to verify
    Check the IP against Google's published list. Google says it is also experimenting with signed requests (Web Bot Auth) for this agent. Google's IP list
    Block Google-Agent
    User-agent: Google-Agent
Disallow: /
    Allow Google-Agent
    User-agent: Google-Agent
Allow: /

    About Google-AgentVerify Google-Agent requests

  • Google-GeminiNotebook

    Google-GeminiNotebook

    Google

    Ignores robots.txtReverse DNS

    Google-GeminiNotebook is the fetcher Google uses to read pages that Gemini Notebook users add as sources.

    Sample user agent
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)
    Robots.txt
    No. Google says user-triggered fetchers generally ignore robots.txt rules. It replaces the Google-NotebookLM token.
    How to verify
    Reverse DNS on google.com or gae.googleusercontent.com, confirmed with a forward lookup. Google publishes IP lists for user-triggered fetchers but does not say which one applies to this fetcher.
    Block Google-GeminiNotebook
    User-agent: Google-GeminiNotebook
Disallow: /
    Allow Google-GeminiNotebook
    User-agent: Google-GeminiNotebook
Allow: /

    About Google-GeminiNotebookVerify Google-GeminiNotebook requests

  • ElevenlabsBot

    ElevenlabsBot

    ElevenLabs

    Follows robots.txtNo way to verify

    ElevenlabsBot is the agent ElevenLabs uses to fetch pages its customers add to their agents' knowledge bases.

    Sample user agent
    ElevenlabsBot/1.0
    Robots.txt
    Yes, according to ElevenLabs.
    How to verify
    ElevenLabs doesn't publish a way to verify it.
    Block ElevenlabsBot
    User-agent: ElevenlabsBot
Disallow: /
    Allow ElevenlabsBot
    User-agent: ElevenlabsBot
Allow: /

    About ElevenlabsBot

  • Diffbot-User

    Diffbot-User

    Diffbot

    Not documentedNo way to verify

    Diffbot-User is the agent Diffbot uses for fetches its users request.

    Sample user agent
    Diffbot does not publish its full user agent strings.
    Robots.txt
    Not documented by the operator. Diffbot does not state this separately for Diffbot-User.
    How to verify
    Diffbot doesn't publish a way to verify it.
    Block Diffbot-User
    User-agent: Diffbot-User
Disallow: /
    Allow Diffbot-User
    User-agent: Diffbot-User
Allow: /

    About Diffbot-User

Classic search

Search engines. Blocking them removes you from search results.

  • Googlebot

    Googlebot

    Google

    Follows robots.txtIP list

    Googlebot is Google's search crawler that finds and indexes pages for Google Search.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
    Robots.txt
    Yes, according to Google. Google says its common crawlers always respect robots.txt rules for automatic crawls.
    How to verify
    Check the IP against Google's published list. Reverse DNS on googlebot.com or google.com, confirmed with a forward lookup. Google's IP list

    Blocking Googlebot removes your pages from Google search results. If you need to, write the rule by hand.

    Allow Googlebot
    User-agent: Googlebot
Allow: /

    About GooglebotVerify Googlebot requests

  • bingbot

    bingbot

    Microsoft

    Not yet verified by usIP list

    bingbot is Microsoft's crawler for Bing search, which also feeds Microsoft Copilot answers.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
    Robots.txt
    Not yet verified by us. Bing's crawler help page could not be read at our review, so we have not confirmed its statement yet.
    How to verify
    Check the IP against Microsoft's published list. Reverse DNS on search.msn.com, confirmed with a forward lookup. Bing has no separate AI token: it controls Copilot use with NOARCHIVE and NOCACHE meta tags. Microsoft's IP list

    Blocking bingbot removes your pages from Microsoft search results. If you need to, write the rule by hand.

    Allow bingbot
    User-agent: bingbot
Allow: /

    About bingbotVerify bingbot requests

  • Applebot

    Applebot

    Apple

    Follows robots.txtIP list

    Applebot is Apple's crawler for search features in Spotlight, Siri and Safari.

    Sample user agent
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
    Robots.txt
    Yes, according to Apple. Apple says it follows your Googlebot rules when Applebot is not named, and does not support Crawl-delay.
    How to verify
    Check the IP against Apple's published list. Reverse DNS on applebot.apple.com, confirmed with a forward lookup. Apple's IP list

    Blocking Applebot removes your pages from Apple search results. If you need to, write the rule by hand.

    Allow Applebot
    User-agent: Applebot
Allow: /

    About ApplebotVerify Applebot requests

  • DuckDuckBot

    DuckDuckBot

    DuckDuckGo

    Follows robots.txtIP list

    DuckDuckBot is DuckDuckGo's crawler for DuckDuckGo search results.

    Sample user agent
    DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
    Robots.txt
    Yes, according to DuckDuckGo.
    How to verify
    Check the IP against DuckDuckGo's published list. DuckDuckGo's IP list

    Blocking DuckDuckBot removes your pages from DuckDuckGo search results. If you need to, write the rule by hand.

    Allow DuckDuckBot
    User-agent: DuckDuckBot
Allow: /

    About DuckDuckBotVerify DuckDuckBot requests

  • YandexBot

    YandexBot

    Yandex

    Partly followsReverse DNS

    YandexBot is Yandex's main indexing crawler for Yandex search.

    Sample user agent
    Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
    Robots.txt
    Partly. Yandex says YandexBot follows the general rules, but some Yandex robots may ignore robots.txt, including rules for User-agent: Yandex.
    How to verify
    Reverse DNS on yandex.ru, yandex.net or yandex.com, confirmed with a forward lookup. Yandex does not publish an IP list.

    Blocking YandexBot removes your pages from Yandex search results. If you need to, write the rule by hand.

    Allow YandexBot
    User-agent: YandexBot
Allow: /

    About YandexBotVerify YandexBot requests

  • Baiduspider

    Baiduspider

    Baidu

    Partly followsReverse DNS

    Baiduspider is Baidu's search crawler that adds pages to the Baidu index.

    Sample user agent
    Baidu publishes the token but not a full user agent string.
    Robots.txt
    Partly. Baidu says Baiduspider obeys robots.txt, but Baiduspider-video does not support it yet and its ads crawlers do not follow it.
    How to verify
    Reverse DNS on baidu.com or baidu.jp, confirmed with a forward lookup.

    Blocking Baiduspider removes your pages from Baidu search results. If you need to, write the rule by hand.

    Allow Baiduspider
    User-agent: Baiduspider
Allow: /

    About BaiduspiderVerify Baiduspider requests

Link preview

Builds the preview card when someone shares your link in a chat or social app.

  • facebookexternalhit

    facebookexternalhit

    Meta

    Partly followsNo way to verify

    facebookexternalhit is the bot Meta uses to build link previews when someone shares your page on Facebook, Instagram or WhatsApp.

    Sample user agent
    facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
    Robots.txt
    Partly. Meta says it might bypass robots.txt when performing security or integrity checks.
    How to verify
    Meta doesn't publish a way to verify it.
    Block facebookexternalhit
    User-agent: facebookexternalhit
Disallow: /
    Allow facebookexternalhit
    User-agent: facebookexternalhit
Allow: /

    About facebookexternalhit

  • Twitterbot

    Twitterbot

    X

    Follows robots.txtNo way to verify

    Twitterbot is the bot X uses to read card markup and build the preview when someone posts your link.

    Sample user agent
    X says it uses the Twitterbot user agent with a version, such as Twitterbot/1.0, but publishes no full string.
    Robots.txt
    Yes, according to X. X says that if a page is blocked, no card is shown.
    How to verify
    X doesn't publish a way to verify it.
    Block Twitterbot
    User-agent: Twitterbot
Disallow: /
    Allow Twitterbot
    User-agent: Twitterbot
Allow: /

    About Twitterbot

  • Slackbot-LinkExpanding

    Slackbot-LinkExpanding

    Slack

    Ignores robots.txtNo way to verify

    Slackbot-LinkExpanding is the bot Slack uses to expand links posted in channels into previews.

    Sample user agent
    Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
    Robots.txt
    No. Slack says it does not currently honor robots.txt, because it acts for a person and does not follow links.
    How to verify
    Slack doesn't publish a way to verify it.
    Block Slackbot-LinkExpanding
    User-agent: Slackbot-LinkExpanding
Disallow: /
    Allow Slackbot-LinkExpanding
    User-agent: Slackbot-LinkExpanding
Allow: /

    About Slackbot-LinkExpanding

  • Discordbot

    Discordbot

    Discord

    Not documentedIP list

    Discordbot is the bot Discord uses to visit a shared link and build its preview.

    Sample user agent
    Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)
    Robots.txt
    Not documented by the operator. Discord's article does not mention robots.txt. It says the bot only visits links shared in messages.
    How to verify
    Check the IP against Discord's published list. Discord's IP list
    Block Discordbot
    User-agent: Discordbot
Disallow: /
    Allow Discordbot
    User-agent: Discordbot
Allow: /

    About DiscordbotVerify Discordbot requests

Other bots

Ads checks, SEO tools and research crawls. Not AI training or AI search.

  • OAI-AdsBot

    OAI-AdsBot

    OpenAI

    Not documentedIP list

    OAI-AdsBot is the bot OpenAI uses to check pages submitted as ads in ChatGPT; OpenAI says it is not used for training.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
    Robots.txt
    Not documented by the operator. OpenAI does not say. It only visits pages submitted as ads.
    How to verify
    Check the IP against OpenAI's published list. OpenAI's IP list
    Block OAI-AdsBot
    User-agent: OAI-AdsBot
Disallow: /
    Allow OAI-AdsBot
    User-agent: OAI-AdsBot
Allow: /

    About OAI-AdsBotVerify OAI-AdsBot requests

  • GoogleOther

    GoogleOther

    Google

    Follows robots.txtIP list

    GoogleOther is the generic crawler Google teams use for research and development crawls.

    Sample user agent
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36
    Robots.txt
    Yes, according to Google. Google says its common crawlers always respect robots.txt rules for automatic crawls.
    How to verify
    Check the IP against Google's published list. Reverse DNS on googlebot.com or google.com, confirmed with a forward lookup. Google's IP list
    Block GoogleOther
    User-agent: GoogleOther
Disallow: /
    Allow GoogleOther
    User-agent: GoogleOther
Allow: /

    About GoogleOtherVerify GoogleOther requests

  • Google-CloudVertexBot

    Google-CloudVertexBot

    Google

    Follows robots.txtIP list

    Google-CloudVertexBot is the crawler Google runs when a site owner asks it to crawl their site for Vertex AI Agents.

    Sample user agent
    Google says the user agent contains Google-CloudVertexBot but does not publish a full string.
    Robots.txt
    Yes, according to Google. Google says it has no effect on Google Search or other products.
    How to verify
    Check the IP against Google's published list. Reverse DNS on googlebot.com or google.com, confirmed with a forward lookup. Google's IP list
    Block Google-CloudVertexBot
    User-agent: Google-CloudVertexBot
Disallow: /
    Allow Google-CloudVertexBot
    User-agent: Google-CloudVertexBot
Allow: /

    About Google-CloudVertexBotVerify Google-CloudVertexBot requests

  • Meta-ExternalAds

    Meta-ExternalAds

    meta-externalads

    Follows robots.txtNo way to verify

    Meta-ExternalAds is Meta's crawler for its ads and business products.

    Sample user agent
    meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
    Robots.txt
    Yes, according to Meta.
    How to verify
    Meta doesn't publish a way to verify it.
    Block Meta-ExternalAds
    User-agent: meta-externalads
Disallow: /
    Allow Meta-ExternalAds
    User-agent: meta-externalads
Allow: /

    About Meta-ExternalAds

  • Diffbot

    Diffbot

    Diffbot

    Partly followsNo way to verify

    Diffbot is Diffbot's crawler for its Knowledge Graph and search products; Diffbot says it is not used for AI training.

    Sample user agent
    Diffbot does not publish its full user agent strings.
    Robots.txt
    Partly. Diffbot says it follows robots.txt and Crawl-delay by default, but this can be overridden by partnership agreements.
    How to verify
    Diffbot doesn't publish a way to verify it.
    Purpose
    Diffbot says it is not used for AI training. Other sources describe it as an AI training crawler.
    Block Diffbot
    User-agent: Diffbot
Disallow: /
    Allow Diffbot
    User-agent: Diffbot
Allow: /

    About Diffbot

  • AhrefsBot

    AhrefsBot

    Ahrefs

    Follows robots.txtIP list

    AhrefsBot is Ahrefs' crawler for its SEO index and the Yep search engine.

    Sample user agent
    Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
    Robots.txt
    Yes, according to Ahrefs. Ahrefs says it respects Allow, Disallow and Crawl-delay, but not for JavaScript rendering.
    How to verify
    Check the IP against Ahrefs's published list. Reverse DNS on ahrefs.com or ahrefs.net, confirmed with a forward lookup. Ahrefs's IP list
    Block AhrefsBot
    User-agent: AhrefsBot
Disallow: /
    Allow AhrefsBot
    User-agent: AhrefsBot
Allow: /

    About AhrefsBotVerify AhrefsBot requests

  • AhrefsSiteAudit

    AhrefsSiteAudit

    Ahrefs

    Partly followsIP list

    AhrefsSiteAudit is Ahrefs' crawler for the site audits its customers run.

    Sample user agent
    Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
    Robots.txt
    Partly. Ahrefs says it follows robots.txt by default, but the verified owner of a site can tell it to ignore the rules.
    How to verify
    Check the IP against Ahrefs's published list. Reverse DNS on ahrefs.com or ahrefs.net, confirmed with a forward lookup. Ahrefs's IP list
    Block AhrefsSiteAudit
    User-agent: AhrefsSiteAudit
Disallow: /
    Allow AhrefsSiteAudit
    User-agent: AhrefsSiteAudit
Allow: /

    About AhrefsSiteAuditVerify AhrefsSiteAudit requests

Training, search and user fetches are different decisions

AI training crawlers collect pages to train models. AI search crawlers index pages so AI answers can cite and link to you. User-triggered fetchers read a page because a person asked an assistant about it. Classic search crawlers build search engine results, and link preview bots build the card when someone shares your link.

The point most people miss: blocking a training crawler such as GPTBot does not remove you from AI answers. Blocking an AI search crawler such as OAI-SearchBot can.

Why some bots can't be verified

A user agent is a label, and anyone can type it. A script can send GPTBot as easily as OpenAI can. Only something an impostor cannot fake proves who sent a request: an IP address in the operator's published list, a DNS name on the operator's domain, or a signed request.

Each entry shows which of these the operator publishes. When it publishes none, the user agent is the only signal you have.

How we keep this list

Every entry is checked against the operator's own documentation, links to that page, and shows the date it was last reviewed. When the documentation doesn't say something, we say “not documented” rather than guess. Where only third-party trackers describe a bot, the entry says so.

The tokens come from the same list Referscope uses in the product. To turn your choices into a file, use the AI Crawler Policy Generator.

Questions about AI crawlers

Which AI crawlers should I block?

If you don't want your pages used to train models, block the AI training crawlers and keep AI search and user-triggered fetches allowed, so AI answers can still cite and link to you. The AI Crawler Policy Generator writes that file for you.

Does blocking Google-Extended hurt my Google ranking?

No. Google Search crawls with Googlebot, which Google-Extended does not affect. But Google-Extended also controls Gemini grounding, so blocking it can keep your pages out of some Gemini answers, not only out of training.

Do AI crawlers obey robots.txt?

The major operators say their crawlers do. Some say fetches a user asked for may not follow it, and some bots ignore it. Each entry shows what the operator documents.

A bot in my logs isn't listed. What is it?

It may be a scraper, a monitoring tool or a crawler we don't list yet. Look up who owns the IP address with a WHOIS or ASN lookup, and don't trust the user agent on its own.

More free tools

AI Crawler Policy Generator

Build a robots.txt that blocks AI training and keeps AI search and Google.

Build your robots.txt

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

All free tools

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner