Skip to content

Crawler

What is Diffbot?

Diffbot is Diffbot's crawler for its Knowledge Graph and search products; Diffbot says it is not used for AI training.

Other botsPartly followsNo way to verify
Operator
Diffbot
Purpose
Other bots. Ads checks, SEO tools and research crawls. Not AI training or AI search.Diffbot says it is not used for AI training. Other sources describe it as an AI training crawler.
robots.txt token
Diffbot
Sample user agent
Diffbot does not publish its full user agent strings.User agents change with versions. Match on the token, not the full string.
Follows robots.txt
Partly. Diffbot says it follows robots.txt and Crawl-delay by default, but this can be overridden by partnership agreements.
How to verify
Diffbot doesn't publish a way to verify it.
Sources
Diffbot documentation
Last reviewed

Allow or block Diffbot

Block Diffbot
User-agent: Diffbot
Disallow: /

Asks Diffbot not to crawl any page. Well-behaved crawlers stop; it does not stop anyone who ignores robots.txt.

Allow Diffbot
User-agent: Diffbot
Allow: /

Lets Diffbot crawl everything. You only need this if a broader rule would otherwise block it.

Should you block it?

Diffbot is not an AI training or AI search crawler. Block it only if you don't want Diffbot's product to read your pages.

Write a full policy with the AI Crawler Policy Generator

Diffbot can't be verified

Diffbot doesn't publish IP ranges, DNS names or signatures for Diffbot. The user agent is the only signal, and anyone can copy it, so treat a request that claims to be Diffbot as unproven.

Questions about Diffbot

Is Diffbot safe to allow?

Diffbot reads public pages; it does not log in or submit forms. Whether to allow it depends on whether you want Diffbot's product to use your pages.

How do I know a request is really Diffbot?

You can't, from the request alone. Diffbot doesn't publish IP ranges, DNS names or signatures for it, so the user agent is the only signal, and anyone can copy that.

How often does Diffbot visit?

It depends on your site; Diffbot doesn't publish a schedule. Referscope shows the real visits.

Related crawlers

All AI crawlers

Free tools for Diffbot

AI Crawler Policy Generator

Build a robots.txt that blocks AI training and keeps AI search and Google.

Build your robots.txt

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner