Skip to content

robots.txt control

What is Google-Extended?

Google-Extended is a robots.txt token Google reads to decide whether your pages may be used for Gemini training and grounding.

AI trainingFollows robots.txtToken only
Operator
Google
Purpose
AI training. Collects pages to train AI models. Blocking it does not remove you from AI answers.Google-Extended also controls Gemini grounding, so blocking it can keep your pages out of some Gemini answers. It does not affect Google Search.
robots.txt token
Google-ExtendedA robots.txt control that never appears in your logs.
Sample user agent
None. No crawler sends this user agent. Google reads the token from robots.txt.
Follows robots.txt
Yes, according to Google. It is a robots.txt control, not a crawler. Blocking it does not affect Google Search.
How to verify
Nothing to verify: no crawler sends this user agent.
Sources
Google documentation
Last reviewed

Allow or block Google-Extended

Block Google-Extended
User-agent: Google-Extended
Disallow: /

Asks Google-Extended not to crawl any page. Well-behaved crawlers stop; it does not stop anyone who ignores robots.txt.

Allow Google-Extended
User-agent: Google-Extended
Allow: /

Lets Google-Extended crawl everything. You only need this if a broader rule would otherwise block it.

Should you block it?

Blocking Google-Extended asks Google not to use new crawls of your pages for training. It doesn't remove you from AI answers. Google-Extended also controls Gemini grounding, so blocking it can keep your pages out of some Gemini answers. It does not affect Google Search.

Write a full policy with the AI Crawler Policy Generator

Google-Extended never appears in your logs

No crawler sends the Google-Extended user agent. Google reads the token from your robots.txt when it decides how pages it already crawls may be used, so there is no request to verify.

Questions about Google-Extended

Is Google-Extended safe to allow?

Google-Extended reads public pages; it does not log in or submit forms. Whether to allow it is about how your content is used, not security.

How do I know a request is really Google-Extended?

No request ever carries Google-Extended: it is a robots.txt token only. Google reads it when deciding how to use pages it already crawls.

How often does Google-Extended visit?

It depends on your site; Google doesn't publish a schedule. Referscope shows the real visits.

Related crawlers

All AI crawlers

Free tools for Google-Extended

AI Crawler Policy Generator

Build a robots.txt that blocks AI training and keeps AI search and Google.

Build your robots.txt

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner