What is generative engine optimization
How optimizing for AI answer engines differs from classic SEO, and what you can measure today instead of guessing at visibility scores.
Learn
Generative engine optimization and AI crawler guides
People usually meet an AI crawler by name in their server logs. This is the hub for understanding those crawlers, verifying them, and measuring the traffic generative engines actually send.
ClaudeBot: what it is and what it does
Why ClaudeBot shows up in your logs, how to recognize it, and how to tell a verified crawl from a spoofed user agent.
GPTBot and how to verify it
How to confirm a request claiming to be GPTBot really is, using published IP ranges and signatures rather than the user-agent string alone.
The AI crawler list
A working reference of the AI crawlers you are most likely to see, from major assistants to search indexers.
How to track ChatGPT referral traffic
Why ChatGPT referrals often show up as Direct in GA4, and how edge analytics attributes them correctly.
AI crawlers and robots.txt
How robots.txt directives apply to AI crawlers, and how to check what your current file actually allows.
Learn more