GPTBot in robots.txt: Which AI Crawlers to Block

If AI search can’t crawl your site, it can’t cite you. That’s the whole story. Your robots.txt is a plain text file at the root of your domain that tells bots where they’re allowed to go. If it tells the AI crawlers to stay out, then ChatGPT, Perplexity, and Google’s AI features never read your pages, and you don’t show up in their answers.

Most people blocking GPTBot think they’re making one decision. They’re making two, and the second one is the expensive one.

The AI crawlers are not one switch

The big AI companies run separate bots for separate jobs, and each has its own name in robots.txt:

Training bots (GPTBot, ClaudeBot, Google-Extended) collect content to train the foundation models. Blocking these keeps your content out of future training data. It does nothing to whether you get cited in live answers.
Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index your content so it can show up in AI search answers with a link back to you. These are the ones that matter for citations.
User fetchers (ChatGPT-User, Claude-User, Perplexity-User) grab a page in real time when a person asks the assistant to look at it.

That distinction is the entire decision. Blocking GPTBot is defensible if you don’t want your work training a model. Blocking OAI-SearchBot or Claude-SearchBot is a different thing entirely: it pulls you out of the citation pool while doing nothing to keep your content out of training.

People block the first and accidentally do the second, because they used a wildcard or copied a robots.txt snippet off some blog. User-agent: * / Disallow: / catches every one of them.

Why a block is fatal rather than costly

Getting cited by an AI engine is a retrieval game. The engine fetches your page, stores it, and pulls a chunk when someone asks a relevant question. The crawl is step one. Hand the crawler a “Disallow” and there is no step two. No fetch, no index, no citation.

This is worth being precise about, because it’s different from most SEO problems. A thin page ranks badly. A slow page ranks badly. A blocked page doesn’t rank at all. You’re not losing on the margin, you’re not in the running.

Which crawlers should you actually block?

Allow the search bots when:

You want AI engines to cite your public content. For most marketing sites, docs, and blogs, that’s the entire point.
The block was an accident. Often it is: a CMS default, an old “block the scrapers” rule, or a wildcard meant to stop one bad bot that caught the good ones too.

Think twice when:

You have a real reason to keep content out of AI. Original research you sell, paywalled material, anything where being summarized for free actually costs you money. That’s a legitimate business call, and no audit can make it for you.
You only want to block training, not search. If your goal is “don’t train on my stuff, but do cite me,” the fix is precise: block GPTBot and Google-Extended, keep OAI-SearchBot and Claude-SearchBot open. Don’t let a blanket rule decide it for you.

The honest tradeoff: blocking AI crawlers does almost nothing to protect your content, because the page is already public and the user fetchers can reach it on demand anyway. What it reliably does is make you invisible in AI answers. For most sites that’s a bad trade.

How to check what you’re blocking

Open yourdomain.com/robots.txt in a browser and read it. You’re looking for three things: a User-agent: * with a broad Disallow, any named AI bot with a Disallow, and any rule you don’t recognize. If you find a Disallow: / under a wildcard, every crawler on this page is blocked right now.

Two traps worth knowing. First, robots.txt is not the only thing that can block a crawler: a firewall rule, a WAF, or a Cloudflare bot setting can block them too, and none of those show up in the file. If your robots.txt looks clean and you’re still invisible, check the layer above it. Second, robots.txt is read on the crawler’s next visit, so a change takes effect when the bots come back, not instantly. Re-crawl and re-indexing take time after that. You make this fix now and see it pay off over weeks.

Once you’ve decided which crawlers belong, the edit itself is small. Get AI Traffic applies it with edit_robots_txt, which narrows the crawler blocks and leaves your private paths and any deliberate training blocks alone.

Frequently asked questions

Does blocking GPTBot stop me from being cited in ChatGPT? No. GPTBot is for training. Citations in ChatGPT search come from OAI-SearchBot. If you want your content out of training but still citable, block GPTBot and leave OAI-SearchBot allowed.

Will unblocking AI crawlers hurt my Google rankings? No. Google’s AI features run on the normal search index that Googlebot builds. Google-Extended only controls training, and Google documents it as a standalone token whose use does not affect inclusion or ranking in Google Search. AI Overviews draw on the live search index, so blocking Google-Extended doesn’t gate them either. The AI search crawlers from OpenAI, Anthropic, and Perplexity are separate from Google entirely.

Do AI crawlers actually obey robots.txt? The automated ones do. OpenAI, Anthropic, and Perplexity all document that their indexing and training crawlers respect robots.txt. The exception is the real-time user fetchers: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity-User behaves the same way, so a Disallow won’t reliably stop on-demand fetches. Anthropic is the outlier here, saying even Claude-User honors robots.txt.

What’s the difference between GPTBot and OAI-SearchBot? GPTBot collects training data for OpenAI’s models. OAI-SearchBot indexes pages so they can appear, with a link, in ChatGPT’s search answers. They are separate user agents and you can allow one while blocking the other.

Is robots.txt the only thing that can block AI crawlers? No. A firewall rule, a WAF, a Cloudflare bot setting, or a server config can block them too, and none of those show up in robots.txt. If you unblock robots.txt and still see nothing, check the layer above it.

GPTBot, ClaudeBot, and the AI Crawlers Your robots.txt Is Blocking

The AI crawlers are not one switch

Why a block is fatal rather than costly

Which crawlers should you actually block?

How to check what you’re blocking

Frequently asked questions

Sources