Your robots.txt Is Telling AI Search to Stay Out

Published on June 08, 2026

If AI search can't crawl your site, it can't cite you. That's the whole story. Your robots.txt is a plain text file at the root of your domain that tells bots where they're allowed to go. If it tells the AI crawlers to stay out, then ChatGPT, Perplexity, and Google's AI features never read your pages, and you don't show up in their answers.

If this action item landed in your audit, the agent found a rule in your robots.txt that blocks one or more AI crawlers. Here's what that means and how to decide whether to fix it.

Why blocking AI crawlers kills AI traffic

Getting cited by an AI engine is a retrieval game. The engine fetches your page, stores it, and pulls a chunk when someone asks a relevant question. The crawl is step one. Hand the crawler a "Disallow" in your robots.txt and there is no step two. No fetch, no index, no citation. You're not ranked low. You're not in the running at all.

Most people get one thing wrong: they treat "AI crawlers" as a single switch. They aren't. The big AI companies now run separate bots for separate jobs, and each one has its own name in robots.txt:

  • Training bots (GPTBot, ClaudeBot, Google-Extended) collect content to train the foundation models. Blocking these keeps your content out of future training data. It does nothing to whether you get cited in live answers.
  • Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index your content so it can show up in AI search answers with a link back to you. These are the ones that matter for citations.
  • User fetchers (ChatGPT-User, Claude-User, Perplexity-User) grab a page in real time when a person asks the assistant to look at it.

That distinction is the entire decision. Blocking GPTBot is defensible if you don't want your work training a model. Blocking OAI-SearchBot or Claude-SearchBot is a different thing entirely: it pulls you out of the citation pool while doing nothing to keep your content out of training. People block the first and accidentally do the second, because they used a wildcard or copied a robots.txt snippet off some blog.

What approving this fix does

Approving this action item tells the agent to edit your robots.txt so the AI search crawlers can read the pages you want cited. In practice that means removing or narrowing the Disallow rules that hit the search and retrieval bots: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and the user fetchers.

The agent does not blindly open everything. It leaves your existing rules for admin paths, carts, and private areas alone. It targets the rules costing you AI visibility, not the ones doing real work.

When to approve, and when to skip

Approve this when:

  • You want AI engines to cite your public content. For most marketing sites, docs, and blogs, that's the entire point.
  • The block was an accident. Often it is: a CMS default, an old "block the scrapers" rule, or a wildcard meant to stop one bad bot that caught the good ones too.

Think twice when:

  • You have a real reason to keep content out of AI. Original research you sell, paywalled material, anything where being summarized for free actually costs you money. That's a legitimate business call, and the audit can't make it for you.
  • You only want to block training, not search. If your goal is "don't train on my stuff, but do cite me," the fix is precise: block GPTBot and Google-Extended, keep OAI-SearchBot and Claude-SearchBot open. Don't let a blanket rule decide it for you.

The honest tradeoff: blocking AI crawlers does almost nothing to protect your content, because the page is already public and the user fetchers can reach it on demand anyway. What it reliably does is make you invisible in AI answers. For most sites that's a bad trade.

How the fix gets applied

Once you approve, the agent edits the robots.txt rules, keeps your existing protections for private paths, and publishes the change through your site's connection. robots.txt is read on the crawler's next visit, so the change takes effect when the bots come back, not instantly. Re-crawl and re-indexing take time after that. You make this fix now and see it pay off over weeks, not minutes.

Frequently asked questions

Does blocking GPTBot stop me from being cited in ChatGPT? No. GPTBot is for training. Citations in ChatGPT search come from OAI-SearchBot. If you want your content out of training but still citable, block GPTBot and leave OAI-SearchBot allowed.

Will unblocking AI crawlers hurt my Google rankings? No. Google's AI features run on the normal search index that Googlebot builds. Google-Extended only controls training, and Google documents it as a standalone token whose use does not affect inclusion or ranking in Google Search. AI Overviews draw on the live search index, so blocking Google-Extended doesn't gate them either. The AI search crawlers from OpenAI, Anthropic, and Perplexity are separate from Google entirely.

Do AI crawlers actually obey robots.txt? The automated ones do. OpenAI, Anthropic, and Perplexity all document that their indexing and training crawlers respect robots.txt. The exception is the real-time user fetchers: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity-User behaves the same way, so a Disallow won't reliably stop on-demand fetches. Anthropic is the outlier here, saying even Claude-User honors robots.txt.

Is robots.txt the only thing that can block AI crawlers? No. A firewall rule, a WAF, a Cloudflare bot setting, or a server config can block them too, and none of those show up in robots.txt. If you unblock robots.txt and still see nothing, check the layer above it.

Sources