Blocking GPTBot is usually discussed as if it were one decision with one consequence: protect your content, or stay visible in ChatGPT. It is not that decision. GPTBot only does one of those jobs, and the sites that disappear from ChatGPT almost always blocked a different bot.
This post sets out what each AI crawler actually does, according to the companies that run them, what blocking each one costs you, and the robots.txt for each choice.
The short answer
Blocking GPTBot keeps your future content out of OpenAI’s training data. It does not remove you from ChatGPT’s answers. OpenAI runs a separate crawler, OAI-SearchBot, for ChatGPT search, and states that “each setting is independent of the others.”
So the real question is not “block GPTBot or not”. It is two questions: do you want your content used to train models, and do you want to be named in AI answers? You can answer them separately, and most businesses should.
Three jobs, and a bot for each
Every major AI company now runs more than one agent. They fall into three jobs, and confusing the jobs is where most of the damage happens.
1. Training crawlers
Collect content that may be used to train future models. Blocking them is a content licensing decision.
- GPTBot (OpenAI): “used to crawl content that may be used in training our generative AI foundation models.”
- ClaudeBot (Anthropic): collects content that could contribute to model training. Blocking it excludes your future content from training datasets.
- Google-Extended (Google): a robots.txt token, not a separate crawler. It controls whether content Google already crawls can be used for Gemini. Google states it “does not impact a site’s inclusion in Google Search.”
- Applebot-Extended (Apple) and CCBot (Common Crawl, an open dataset many models are trained on).
2. Search crawlers
Build the index an assistant searches when it answers a question. Blocking these is what removes you from answers.
- OAI-SearchBot (OpenAI): “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.”
- Claude-SearchBot (Anthropic): indexes content to improve Claude’s search results.
- PerplexityBot (Perplexity): surfaces and links sites in Perplexity results. Perplexity says it is not used to train foundation models.
- Googlebot: the ordinary Google Search crawler, which also serves AI Overviews and AI Mode. There is no robots.txt line that removes you from AI Overviews while keeping you in Search.
3. User-initiated fetchers
Visit a page because a person asked the assistant to read it, right now.
- ChatGPT-User (OpenAI). OpenAI notes that because these requests are user-initiated, “robots.txt rules may not apply.”
- Claude-User (Anthropic). Anthropic says it honours robots.txt; blocking it stops Claude reading your page when a user asks.
- Perplexity-User (Perplexity), which Perplexity says “generally ignores robots.txt rules.”
The case for blocking GPTBot
If your content is the product, blocking the training crawlers is a reasonable commercial decision:
- Publishers and paid research. A model that has absorbed your archive can answer questions your subscribers pay you to answer.
- Courses and proprietary methods. The value is the material itself, not the traffic it brings.
- Negotiating position. Some publishers block training crawlers while they negotiate licensing deals. Content that is already freely crawled is harder to charge for.
Two limits are worth knowing before you rely on it. robots.txt applies to future crawling, not to what was collected before you added the line. And it is a request, not an enforcement mechanism: the named crawlers say they honour it, but anything that does not has to be blocked at your server or CDN.
The case against
If you sell something other than your content, the argument mostly runs the other way. Your pages exist to be found, and a model that has read them knows what you do, who you compete with and which questions you answer.
How much training data influences what an assistant says about a brand is not something anyone outside the AI companies can measure precisely. What is clear is that blocking GPTBot buys a software company, an agency or a shop very little. The content you would be protecting is marketing, and marketing is meant to be read.
For most businesses the sensible default is: leave the training crawlers open unless you have a specific reason, and never block the search crawlers.
The mistake that actually costs visibility
The sites that vanish from AI answers rarely made a considered decision. They made one of these:
- Blocking every AI bot with one toggle. Cloudflare and most CDNs offer an “block AI crawlers” switch. In most setups it blocks search and user-fetch agents along with the training ones, which is usually more than the site owner intended.
- Copying a robots.txt list from an article. Many “block AI bots” lists include OAI-SearchBot and PerplexityBot next to GPTBot, with no explanation that those two are search crawlers.
- Blocking Google-Extended to escape AI Overviews. It does not work. AI Overviews come from Googlebot. The site loses Gemini visibility and gets nothing in return.
- A leftover
Disallow: /underUser-agent: *from a staging site. That blocks every well-behaved crawler, AI or not.
The robots.txt for each choice
robots.txt lives at the root of your domain (yoursite.com/robots.txt). Each block names a user agent and what it may not fetch. Pick the option that matches your decision.
Option A: stay in AI answers, keep content out of training
The most common sensible choice for a publisher that still wants to be found.
# Training crawlers: blocked User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / # Search crawlers: allowed User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: /
Google-Extended is left out on purpose. Blocking it also affects how Gemini can use your content when answering, so it is not a pure training opt-out. Add it only if you accept that trade.
Option B: out of AI completely
Only if you genuinely want to be absent from AI assistants. Remember that ChatGPT-User and Perplexity-User may still fetch a page a user asks for; stopping those needs a server or CDN rule.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: /
Option C: fully open
No AI-specific lines at all. If your robots.txt does not mention these agents and has no blanket Disallow: /, every crawler may read you. For most businesses that sell something other than content, this is the right default.
Check what your site does today
Before changing anything, find out what your current file actually says. Many sites are blocking a search crawler they have never heard of, through a CMS plugin or a CDN setting nobody remembers turning on.
Our free AI crawler checker reads your robots.txt and shows every AI agent it allows or blocks, with the line that decided it and what that block costs. If you are staying open, it is also worth publishing an llms.txt file: it costs nothing, though no major AI company has committed to reading it yet.
Crawler access is one of the 31 checks in the CoreCited site audit, so a plugin update that quietly blocks OAI-SearchBot shows up the week it happens, not the quarter you notice the traffic.
The summary
Blocking GPTBot is a licensing decision. It protects future content from training and costs you nothing in ChatGPT search. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot is a visibility decision, and it removes you from those answers completely. Decide the two separately, write the file to match, and check it after every CMS or CDN change.
Questions people ask
What is GPTBot?
OpenAI's training crawler. It collects public web content that may be used to train OpenAI's foundation models. It is one of several OpenAI agents, and the only one whose job is training.
If I block GPTBot, will ChatGPT stop mentioning my brand?
No. ChatGPT's search answers are built from what OAI-SearchBot indexes, and OpenAI states that each of its crawler settings is independent. Blocking GPTBot keeps new content out of training data; it does not remove you from ChatGPT search.
Does blocking GPTBot remove content OpenAI already collected?
No. robots.txt governs future crawling. It is not a deletion request, and nothing in OpenAI's crawler documentation describes it as one.
Does blocking Google-Extended take me out of AI Overviews?
No. AI Overviews are served by Googlebot, the same crawler as Google Search. Google states that Google-Extended does not affect a site's inclusion in Search. Blocking it only affects Gemini training and grounding.
Do AI crawlers actually obey robots.txt?
The named training and search crawlers from OpenAI, Anthropic, Google and Perplexity say they do. The user-initiated fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. To stop those, block at your server or CDN.