Skip to content

Crawler

What is AhrefsSiteAudit?

AhrefsSiteAudit is Ahrefs' crawler for the site audits its customers run.

Other botsPartly followsIP list
Operator
Ahrefs
Purpose
Other bots. Ads checks, SEO tools and research crawls. Not AI training or AI search.
robots.txt token
AhrefsSiteAudit
Sample user agent
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)User agents change with versions. Match on the token, not the full string.
Follows robots.txt
Partly. Ahrefs says it follows robots.txt by default, but the verified owner of a site can tell it to ignore the rules.
How to verify
Check the IP against Ahrefs's published list.Reverse DNS on ahrefs.com or ahrefs.net, confirmed with a forward lookup.
Sources
Ahrefs documentationAhrefs IP list
Last reviewed

Allow or block AhrefsSiteAudit

Block AhrefsSiteAudit
User-agent: AhrefsSiteAudit
Disallow: /

Asks AhrefsSiteAudit not to crawl any page. Well-behaved crawlers stop; it does not stop anyone who ignores robots.txt.

Allow AhrefsSiteAudit
User-agent: AhrefsSiteAudit
Allow: /

Lets AhrefsSiteAudit crawl everything. You only need this if a broader rule would otherwise block it.

Should you block it?

AhrefsSiteAudit is not an AI training or AI search crawler. Block it only if you don't want Ahrefs's product to read your pages.

Write a full policy with the AI Crawler Policy Generator

How to verify a AhrefsSiteAudit request

  1. Find the IP address of the request in your server or CDN logs.
  2. Get Ahrefs's published list (Ahrefs IP list).
  3. Check whether the IP falls inside one of the ranges. If it doesn't, the request is not from AhrefsSiteAudit, whatever its user agent says.

Or with reverse DNS

  1. Run a reverse DNS lookup on the IP (host 192.0.2.1, with the IP from your log).
  2. Check the hostname ends in ahrefs.com or ahrefs.net.
  3. Run a forward lookup on that hostname and check it returns the same IP. Both steps must pass.
Test AhrefsSiteAudit requests

Questions about AhrefsSiteAudit

Is AhrefsSiteAudit safe to allow?

AhrefsSiteAudit reads public pages; it does not log in or submit forms. Whether to allow it depends on whether you want Ahrefs's product to use your pages.

How do I know a request is really AhrefsSiteAudit?

Check the request against what Ahrefs publishes: Check the IP against Ahrefs's published list. Reverse DNS on ahrefs.com or ahrefs.net, confirmed with a forward lookup. The user agent alone proves nothing.

How often does AhrefsSiteAudit visit?

It depends on your site; Ahrefs doesn't publish a schedule. Referscope shows the real visits.

Related crawlers

All AI crawlers

Free tools for AhrefsSiteAudit

AI Crawler Policy Generator

Build a robots.txt that blocks AI training and keeps AI search and Google.

Build your robots.txt

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner