Skip to content

AI crawler

What is YouBot?

YouBot is You.com's crawler that powers You.com search and its AI answers.

AI search and answersFollows robots.txtSigned requests
Operator
You.com
Purpose
AI search and answers. Indexes pages so AI answers can cite and link to you. Blocking it can remove you from those answers.
robots.txt token
YouBot
Sample user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36User agents change with versions. Match on the token, not the full string.
Follows robots.txt
Yes, according to You.com. You.com says it respects Crawl-delay and caches robots.txt for 30 minutes.
How to verify
You.com signs its requests (checking a signature needs the request headers, not a log line).Reverse DNS on search.you.com, confirmed with a forward lookup.Check the IP against You.com's published list.You.com publishes a single range, 68.67.112.0/24, on its documentation page.
Sources
You.com documentationYou.com IP list
Last reviewed

Allow or block YouBot

Block YouBot
User-agent: YouBot
Disallow: /

Asks YouBot not to crawl any page. Well-behaved crawlers stop; it does not stop anyone who ignores robots.txt.

Allow YouBot
User-agent: YouBot
Allow: /

Lets YouBot crawl everything. You only need this if a broader rule would otherwise block it.

Should you block it?

Keep YouBot allowed if you want to appear, with a link, in You.com's answers. Blocking it is the fastest way to disappear from them.

Write a full policy with the AI Crawler Policy Generator

How to verify a YouBot request

  1. Find the IP address of the request in your server or CDN logs.
  2. Get You.com's published list (You.com IP list).
  3. Check whether the IP falls inside one of the ranges. If it doesn't, the request is not from YouBot, whatever its user agent says.

Or with reverse DNS

  1. Run a reverse DNS lookup on the IP (host 192.0.2.1, with the IP from your log).
  2. Check the hostname ends in search.you.com.
  3. Run a forward lookup on that hostname and check it returns the same IP. Both steps must pass.

You.com also signs its requests. Checking a signature needs the request headers, not a log line; see the You.com documentation.

You.com publishes a single range, 68.67.112.0/24, on its documentation page.

Test YouBot requests

Questions about YouBot

Is YouBot safe to allow?

YouBot reads public pages; it does not log in or submit forms. Whether to allow it depends on whether you want You.com's product to use your pages.

How do I know a request is really YouBot?

Check the request against what You.com publishes: You.com signs its requests (checking a signature needs the request headers, not a log line). Reverse DNS on search.you.com, confirmed with a forward lookup. Check the IP against You.com's published list. The user agent alone proves nothing.

How often does YouBot visit?

It depends on your site; You.com doesn't publish a schedule. Referscope shows the real visits.

Related crawlers

All AI crawlers

Free tools for YouBot

AI Crawler Policy Generator

Build a robots.txt that blocks AI training and keeps AI search and Google.

Build your robots.txt

AI Bot Verification Tester

Paste an IP and user agent from your logs and see if the bot is real.

Verify a bot

robots.txt Checker

See which AI crawlers your robots.txt allows or blocks.

Check robots.txt

Less guessing. Clearer growth.

Connect your site today and see your first verified AI crawler by tomorrow.

Explore ReferscopeBecome a design partner