Skip to content

Free tool

Write a robots.txt for AI crawlers

Choose which AI crawlers may read your site: keep AI search and Google, block training, or set each bot yourself. Then copy or download the file.

Free. No account needed. Nothing leaves your browser: the file is built on this page.

Start from a preset

Preview: 15 blocked. Jump to your robots.txt

Choices by purpose

  • AI training

    Collects pages to train AI models. Blocking it does not remove you from AI answers.

  • AI search and answers

    Indexes pages so AI answers can cite and link to you. Blocking it can remove you from those answers.

  • User-triggered fetch

    Fetches a page when a person asks an assistant about it.

  • Link preview

    Builds the preview card when someone shares your link in a chat or social app.

  • Other bots

    Ads checks, SEO tools and research crawls. Not AI training or AI search.

  • Classic search

    Allowed. Blocking Googlebot or Bingbot removes you from search results, so this generator doesn't do it unless you choose a crawler below.

Set each crawler

AI training: 15 crawlers, 15 blocked

  • GPTBot

    OpenAI

    GPTBot rule

  • ClaudeBot

    Anthropic

    ClaudeBot rule

  • anthropic-ai

    Anthropic

    anthropic-ai rule

  • Google-Extended

    Google

    No crawler sends this. Google reads it as a training opt-out.

    Google-Extended also controls Gemini grounding, so blocking it can keep your pages out of some Gemini answers. It does not affect Google Search.

    Google-Extended rule

  • Applebot-Extended

    Apple

    No crawler sends this. Apple reads it as a training opt-out.

    Applebot-Extended rule

  • CCBot

    Common Crawl

    CCBot rule

  • Meta-ExternalAgent

    Meta

    Meta-ExternalAgent rule

  • FacebookBot

    Meta

    FacebookBot rule

  • Bytespider

    ByteDance

    Bytespider rule

  • Amazonbot

    Amazon

    Amazonbot rule

  • MistralAI-Training

    Mistral

    MistralAI-Training rule

  • ElevenlabsCrawler

    ElevenLabs

    ElevenlabsCrawler rule

  • cohere-ai

    Cohere

    cohere-ai rule

  • cohere-training-data-crawler

    Cohere

    cohere-training-data-crawler rule

  • Timpibot

    Timpi

    Timpibot rule
User-triggered fetch: 10 crawlers, 0 blocked

  • ChatGPT-User

    OpenAI

    ChatGPT-User rule

  • Claude-User

    Anthropic

    Claude-User rule

  • Perplexity-User

    Perplexity

    Perplexity-User rule

  • Meta-ExternalFetcher

    Meta

    Meta-ExternalFetcher rule

  • MistralAI-User

    Mistral

    MistralAI-User rule

  • Amzn-User

    Amazon

    Amzn-User rule

  • Google-Agent

    Google

    Google-Agent rule

  • Google-GeminiNotebook

    Google

    Google-GeminiNotebook rule

  • ElevenlabsBot

    ElevenLabs

    ElevenlabsBot rule

  • Diffbot-User

    Diffbot

    Diffbot-User rule
Link preview: 4 crawlers, 0 blocked

  • facebookexternalhit

    Meta

    facebookexternalhit rule

  • Twitterbot

    X

    Twitterbot rule

  • Slackbot-LinkExpanding

    Slack

    Slackbot-LinkExpanding rule

  • Discordbot

    Discord

    Discordbot rule
Other bots: 7 crawlers, 0 blocked

  • OAI-AdsBot

    OpenAI

    OAI-AdsBot rule

  • GoogleOther

    Google

    GoogleOther rule

  • Google-CloudVertexBot

    Google

    Google-CloudVertexBot rule

  • Meta-ExternalAds

    Meta

    Meta-ExternalAds rule

  • Diffbot

    Diffbot

    Diffbot rule

  • AhrefsBot

    Ahrefs

    AhrefsBot rule

  • AhrefsSiteAudit

    Ahrefs

    AhrefsSiteAudit rule
Merge with your current robots.txt

Paste the whole file. We keep your own rules and replace only the AI crawler groups. It stays in your browser.

Your robots.txt

15 crawlers blocked. Classic search and AI search allowed. 23 lines.

15 crawlers blocked. Classic search and AI search allowed. 23 lines.

Allowed crawlers get no lines: they follow your rules for User-agent: *.

robots.txt preview
# BEGIN AI crawler policy
# robots.txt is a request. Well-behaved crawlers follow it; it does not block anyone.
# AI training crawlers: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: ElevenlabsCrawler
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Timpibot
Disallow: /
# END AI crawler policy

User-agent: *
Allow: /

robots.txt is a request. Well-behaved crawlers follow it; it does not stop the rest.

Training, search and user fetches: what each choice does

AI training crawlers collect pages to train models. Blocking them keeps new crawls of your pages out of training data, and does not remove you from AI answers.

AI search crawlers index pages so assistants like ChatGPT search and Perplexity can cite and link to you. User-triggered fetchers read a page because a person asked about it. Blocking either can take you out of AI answers.

The AI Crawler Directory lists every crawler here, with what its operator documents.

Why we don't add Allow lines for every crawler

A crawler with its own group follows only that group and ignores User-agent: *. So an explicit User-agent: GPTBot / Allow: / group quietly drops any Disallow: /admin/ you wrote under * for that crawler.

That is why allowed crawlers get no lines by default: they keep following your * rules. If your * group blocks everything, tick “Name allowed AI crawlers explicitly” and we write one group for them that copies your * Disallow paths and ends with Allow: /.

What robots.txt can't do

robots.txt is a request. Well-behaved crawlers follow it; a bot that ignores it, or pretends to be another crawler, is not stopped by it. To stop a bot that ignores robots.txt, block it at your CDN or firewall.

It also only affects future crawls. Pages collected before the change stay collected. To find out which crawlers actually follow your rules, you need server-side data on who visits.

Questions about robots.txt for AI

Will blocking AI training crawlers hurt my SEO?

Not your Google Search ranking: Google Search uses Googlebot, and this generator never blocks it unless you choose to. Google-Extended, which the recommended preset blocks, also controls Gemini grounding, so it can keep your pages out of some Gemini answers.

Is blocking AI training the same as opting out of AI answers?

No. AI answers use search and user-triggered crawlers, which the recommended preset keeps allowed.

Does it block content that was already collected?

No. robots.txt affects future crawls only.

Is my robots.txt sent anywhere?

No. The file is built in your browser, and a file you paste to merge stays in your browser too.

More free tools

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

AI Crawler Directory

Every major AI crawler: who runs it, what it collects, how to block it.

Browse crawlers

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

All free tools

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner