Can AI crawlers read your site?

The check runs five tests on the page you enter. Each one maps to a real reason AI crawlers miss content.

• Crawler access. Whether your robots.txt lets AI crawlers such as GPTBot, ClaudeBot and PerplexityBot fetch your pages. A blanket Disallow: / blocks all of them, even if you only meant to stop spam bots.
• Structured data. Whether the page carries JSON-LD (or other schema markup) that says what it is about and who made it. Without it, crawlers infer everything from body text.
• Meta description. The short summary crawlers read first.
• H1 heading. The main heading that names what the page is about.
• Viewport tag. The basic mobile setup search engines expect.

We also show two hints that are not counted as checks: a rough content signal and whether the HTML mentions responsive layout.

The readiness score combines these checks into a rough guide. It is not a ranking and does not predict whether an AI answer will cite you.

Most founders do not know they are blocking AI crawlers, because those crawlers did not exist when the site was built. Three patterns explain most low results:

A robots.txt that blocks everything. A Disallow: / under User-agent: *, added to stop spam crawlers, also stops GPTBot and ClaudeBot.

No structured data anywhere. Without schema markup, crawlers have to guess what the product does and can file the page under the wrong topic.

A missing or generic meta description. "Welcome to our site" tells a crawler nothing about the page.

Fixing robots.txt

Open yoursite.com/robots.txt and add a group for each AI crawler you want to allow, above any wildcard rules:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Adding structured data

Add a JSON-LD block to the page <head> with at least a WebSite or SoftwareApplication type. Google's Rich Results Test can validate it.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Your product",
  "applicationCategory": "BusinessApplication",
  "url": "https://yoursite.com"
}
</script>

Writing a useful meta description

One sentence, under about 155 characters, that answers "what is this page about?" Skip openers like "Learn more about" and state the thing plainly.