Skip to content

AI crawler

What is Meta-WebIndexer?

Meta-WebIndexer is Meta's crawler that indexes pages so Meta AI can cite them in its answers.

AI search and answersFollows robots.txtNo way to verify
Operator
Meta
Purpose
AI search and answers. Indexes pages so AI answers can cite and link to you. Blocking it can remove you from those answers.
robots.txt token
meta-webindexer
Sample user agent
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)User agents change with versions. Match on the token, not the full string.
Follows robots.txt
Yes, according to Meta.
How to verify
Meta doesn't publish a way to verify it.
Sources
Meta documentation
Last reviewed

Allow or block Meta-WebIndexer

Block Meta-WebIndexer
User-agent: meta-webindexer
Disallow: /

Asks Meta-WebIndexer not to crawl any page. Well-behaved crawlers stop; it does not stop anyone who ignores robots.txt.

Allow Meta-WebIndexer
User-agent: meta-webindexer
Allow: /

Lets Meta-WebIndexer crawl everything. You only need this if a broader rule would otherwise block it.

Should you block it?

Keep Meta-WebIndexer allowed if you want to appear, with a link, in Meta's answers. Blocking it is the fastest way to disappear from them.

Write a full policy with the AI Crawler Policy Generator

Meta-WebIndexer can't be verified

Meta doesn't publish IP ranges, DNS names or signatures for Meta-WebIndexer. The user agent is the only signal, and anyone can copy it, so treat a request that claims to be Meta-WebIndexer as unproven.

Questions about Meta-WebIndexer

Is Meta-WebIndexer safe to allow?

Meta-WebIndexer reads public pages; it does not log in or submit forms. Whether to allow it depends on whether you want Meta's product to use your pages.

How do I know a request is really Meta-WebIndexer?

You can't, from the request alone. Meta doesn't publish IP ranges, DNS names or signatures for it, so the user agent is the only signal, and anyone can copy that.

How often does Meta-WebIndexer visit?

It depends on your site; Meta doesn't publish a schedule. Referscope shows the real visits.

Related crawlers

All AI crawlers

Free tools for Meta-WebIndexer

AI Crawler Policy Generator

Build a robots.txt that blocks AI training and keeps AI search and Google.

Build your robots.txt

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner