Skip to content

AI crawler

What is CCBot?

CCBot is Common Crawl's crawler that builds an open web dataset widely used to train AI models.

AI trainingFollows robots.txtIP list
Operator
Common Crawl
Purpose
AI training. Collects pages to train AI models. Blocking it does not remove you from AI answers.
robots.txt token
CCBot
Sample user agent
CCBot/2.0 (https://commoncrawl.org/faq/)User agents change with versions. Match on the token, not the full string.
Follows robots.txt
Yes, according to Common Crawl.
How to verify
Check the IP against Common Crawl's published list.Reverse DNS on crawl.commoncrawl.org, confirmed with a forward lookup.Reverse DNS works for IPv4 addresses only.
Sources
Common Crawl documentationCommon Crawl IP list
Last reviewed

Allow or block CCBot

Block CCBot
User-agent: CCBot
Disallow: /

Asks CCBot not to crawl any page. Well-behaved crawlers stop; it does not stop anyone who ignores robots.txt.

Allow CCBot
User-agent: CCBot
Allow: /

Lets CCBot crawl everything. You only need this if a broader rule would otherwise block it.

Should you block it?

Blocking CCBot asks Common Crawl not to use new crawls of your pages for training. It doesn't remove you from AI answers.

Write a full policy with the AI Crawler Policy Generator

How to verify a CCBot request

  1. Find the IP address of the request in your server or CDN logs.
  2. Get Common Crawl's published list (Common Crawl IP list).
  3. Check whether the IP falls inside one of the ranges. If it doesn't, the request is not from CCBot, whatever its user agent says.

Or with reverse DNS

  1. Run a reverse DNS lookup on the IP (host 192.0.2.1, with the IP from your log).
  2. Check the hostname ends in crawl.commoncrawl.org.
  3. Run a forward lookup on that hostname and check it returns the same IP. Both steps must pass.

Reverse DNS works for IPv4 addresses only.

Test CCBot requests

Questions about CCBot

Is CCBot safe to allow?

CCBot reads public pages; it does not log in or submit forms. Whether to allow it is about how your content is used, not security.

How do I know a request is really CCBot?

Check the request against what Common Crawl publishes: Check the IP against Common Crawl's published list. Reverse DNS on crawl.commoncrawl.org, confirmed with a forward lookup. The user agent alone proves nothing.

How often does CCBot visit?

It depends on your site; Common Crawl doesn't publish a schedule. Referscope shows the real visits.

Related crawlers

All AI crawlers

Free tools for CCBot

AI Crawler Policy Generator

Build a robots.txt that blocks AI training and keeps AI search and Google.

Build your robots.txt

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner