Is your robots.txt blocking AI crawlers?

A robots.txt file is a plain text file at the root of your domain (for example yourdomain.com/robots.txt). Crawlers fetch it before anything else. If the file says a crawler is not allowed, a well-behaved crawler stops there.

The file is made of groups. Each group starts with one or more User-agent: lines naming crawlers, followed by Allow: and Disallow: rules. User-agent: * applies to every crawler that does not have a group of its own.

When a crawler has its own group, it follows only that group. The two groups do not merge, so you can block everything under * and still allow a named AI crawler in its own group.

Inside a group the most specific rule wins. Allow: /blog/ beats Disallow: / for pages under /blog/, because it is longer. An empty Disallow: line means nothing is disallowed.

Each AI company uses its own user-agent name, and a rule only applies when the name matches. These are the names we check, the same list Referscope uses in the product:

A catch-all Disallow with no named groups. User-agent: * followed by Disallow: / blocks every crawler, AI and search engines included. It is a common setting for staging sites that is easy to leave in place after launch.

An Allow rule in the wrong group. Adding Allow: / under User-agent: * does not help a crawler that has its own group, because that crawler ignores the * group entirely.

A misspelled crawler name. A group only applies when the user-agent name matches the crawler. Copy the names from the table above rather than typing them from memory.

Add these groups above your existing rules to keep AI search and answer crawlers allowed:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-User
Allow: /

# Your existing rules below this line
User-agent: *
Disallow: /admin/

Keeping private sections out

To allow a crawler but keep it out of account pages, add path rules to its group:

User-agent: GPTBot
Allow: /
Disallow: /dashboard/
Disallow: /settings/