Skip to content

AI crawler

What is Amazonbot?

Amazonbot is Amazon's crawler that improves its products and, Amazon says, may be used to train Amazon AI models.

AI trainingFollows robots.txtIP list
Operator
Amazon
Purpose
AI training. Collects pages to train AI models. Blocking it does not remove you from AI answers.
robots.txt token
Amazonbot
Sample user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36User agents change with versions. Match on the token, not the full string.
Follows robots.txt
Yes, according to Amazon. Amazon says it does not support Crawl-delay and caches robots.txt for up to 30 days.
How to verify
Check the IP against Amazon's published list.Amazon publishes its list as a web page of single IP addresses.
Sources
Amazon documentationAmazon IP list
Last reviewed

Allow or block Amazonbot

Block Amazonbot
User-agent: Amazonbot
Disallow: /

Asks Amazonbot not to crawl any page. Well-behaved crawlers stop; it does not stop anyone who ignores robots.txt.

Allow Amazonbot
User-agent: Amazonbot
Allow: /

Lets Amazonbot crawl everything. You only need this if a broader rule would otherwise block it.

Should you block it?

Blocking Amazonbot asks Amazon not to use new crawls of your pages for training. It doesn't remove you from AI answers, and it doesn't affect Amzn-SearchBot.

Write a full policy with the AI Crawler Policy Generator

How to verify a Amazonbot request

  1. Find the IP address of the request in your server or CDN logs.
  2. Get Amazon's published list (Amazon IP list).
  3. Check whether the IP falls inside one of the ranges. If it doesn't, the request is not from Amazonbot, whatever its user agent says.

Amazon publishes its list as a web page of single IP addresses.

Questions about Amazonbot

Is Amazonbot safe to allow?

Amazonbot reads public pages; it does not log in or submit forms. Whether to allow it is about how your content is used, not security.

How do I know a request is really Amazonbot?

Check the request against what Amazon publishes: Check the IP against Amazon's published list. The user agent alone proves nothing.

How often does Amazonbot visit?

It depends on your site; Amazon doesn't publish a schedule. Referscope shows the real visits.

Related crawlers

All AI crawlers

Free tools for Amazonbot

AI Crawler Policy Generator

Build a robots.txt that blocks AI training and keeps AI search and Google.

Build your robots.txt

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner