Skip to content

AI crawler

What is Google-Agent?

Google-Agent is the agent Google uses when a person asks a Google AI agent to act on a page.

User-triggered fetchIgnores robots.txtIP list
Operator
Google
Purpose
User-triggered fetch. Fetches a page when a person asks an assistant about it.
robots.txt token
Google-Agent
Sample user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36User agents change with versions. Match on the token, not the full string.
Follows robots.txt
No. Google says user-triggered fetchers generally ignore robots.txt rules.
How to verify
Check the IP against Google's published list.Google says it is also experimenting with signed requests (Web Bot Auth) for this agent.
Sources
Google documentationGoogle IP list
Last reviewed

Allow or block Google-Agent

Block Google-Agent
User-agent: Google-Agent
Disallow: /

Asks Google-Agent not to crawl any page. Well-behaved crawlers stop; it does not stop anyone who ignores robots.txt.

Allow Google-Agent
User-agent: Google-Agent
Allow: /

Lets Google-Agent crawl everything. You only need this if a broader rule would otherwise block it.

Should you block it?

Google-Agent only visits when a person asks Google's assistant about your page. Blocking it means the assistant can't read the page that person asked about.

Write a full policy with the AI Crawler Policy Generator

How to verify a Google-Agent request

  1. Find the IP address of the request in your server or CDN logs.
  2. Get Google's published list (Google IP list).
  3. Check whether the IP falls inside one of the ranges. If it doesn't, the request is not from Google-Agent, whatever its user agent says.

Google says it is also experimenting with signed requests (Web Bot Auth) for this agent.

Test Google-Agent requests

Questions about Google-Agent

Is Google-Agent safe to allow?

Google-Agent reads public pages; it does not log in or submit forms. Whether to allow it depends on whether you want Google's product to use your pages.

How do I know a request is really Google-Agent?

Check the request against what Google publishes: Check the IP against Google's published list. The user agent alone proves nothing.

How often does Google-Agent visit?

It depends on your site; Google doesn't publish a schedule. Referscope shows the real visits.

Related crawlers

All AI crawlers

Free tools for Google-Agent

AI Crawler Policy Generator

Build a robots.txt that blocks AI training and keeps AI search and Google.

Build your robots.txt

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner