Are you blocking AI by accident?
We read your robots.txt and show you every AI crawler it allows or blocks — with the line that decided it, and what blocking each one actually costs you.
Free · no signup · any domain, not just your own
“Blocking AI” is three different decisions, and most sites only meant one of them.
OpenAI alone runs three crawlers with three different jobs. A site that blocks “OpenAI” by pattern-matching the name usually blocks all three — giving up its place in ChatGPT's answers to avoid being used as training data.
Training crawlers
7Collect content to train future models. GPTBot, ClaudeBot, Google-Extended, CCBot.
Blocking costs you nothing in visibility. It is a content-licensing decision — yours to make either way.
Search crawlers
3Build the index the assistant searches when it answers. OAI-SearchBot, Claude-SearchBot, PerplexityBot.
Blocking removes you from that assistant's answers entirely. This is almost never what someone intends.
User-fetch crawlers
3Open your page when a person's question needs it. ChatGPT-User, Claude-User, Perplexity-User.
Blocking stops someone who explicitly asked about you from being shown your page.
Blocking Google-Extended does not remove you from AI Overviews.
This is the single most expensive misunderstanding in the area, and a lot of sites have already paid for it. AI Overviews are served by Googlebot — the same crawler that powers Google Search. There is no directive that removes you from one and leaves you in the other.
Controls whether your content is used for Gemini training and grounding. That is all it controls.
Block it and you lose Gemini visibility. Your AI Overviews position does not change by a single result.
Serves Google Search and AI Overviews. One crawler, both surfaces.
Block it and you leave AI Overviews — along with the entire Google index. Almost nobody wants that trade.
This is also why we track AI Overviews as a separate surface in the product rather than folding it into “Google”. It behaves differently from Gemini, it is governed by a different crawler, and appearing in one tells you nothing about the other.
It depends what you sell.
We are not going to tell you to open everything. For some businesses the content is the product, and letting a model absorb it for free is a genuine loss. The distinction that matters is which crawler you are turning away.
If your content is the product
A publisher, a paid course, a research archive. Blocking the training crawlers is a reasonable commercial decision and costs you no visibility — but keep the search and user-fetch crawlers open, or you vanish from the assistants your readers are already using.
If your content exists to be found
A SaaS, an agency, a local business, a shop. Your content was written so people would find you. Blocking training crawlers gains you nothing you were selling, and blocking search crawlers costs you the thing you wrote it for.
If you inherited the file
A wildcard block that names no AI crawler individually is usually a staging rule or a CMS default that shipped to production. Check when it was last edited before assuming it was a decision.
If you are using a CDN toggle
Cloudflare and others offer a one-click AI-crawler block. It is all-or-nothing in most implementations — it will take out the search and user-fetch crawlers with the training ones, which is more than most people intend.
Questions about AI crawlers
Does blocking Google-Extended remove me from AI Overviews?+
No. This is the most costly misunderstanding in the whole area. AI Overviews are served by Googlebot — the same crawler that powers Google Search — so there is no robots.txt directive that removes you from AI Overviews while keeping you in Search. Google-Extended only controls whether your content is used for Gemini training and grounding. Blocking it costs you Gemini visibility and changes nothing about AI Overviews.
What is the difference between GPTBot and OAI-SearchBot?+
GPTBot collects content to train future OpenAI models. OAI-SearchBot builds the index ChatGPT searches when it answers a question. Blocking GPTBot is a content-licensing decision and costs you no visibility. Blocking OAI-SearchBot removes you from ChatGPT's answers. Many sites block both while intending only the first.
Should I block AI crawlers?+
It depends on what you sell. If your content IS the product — a publisher, a course, a research archive — blocking the training crawlers is a reasonable commercial decision. If you sell something else and your content exists to be found, blocking them costs you the visibility you wrote it for. Either way, keep the search and user-fetch crawlers open unless you genuinely want to be absent from those assistants.
I have no robots.txt at all. Is that bad?+
Not for AI visibility. No file means no prohibitions, so every crawler may read you — which is the most open position you can be in. A robots.txt is worth adding if you want to exclude something specific, or to point crawlers at your sitemap.
Do AI crawlers actually obey robots.txt?+
The named ones from OpenAI, Anthropic, Perplexity, Google and Apple publish their agent names and state that they honour it, and independent testing has broadly borne that out. It is not an enforcement mechanism though — it is a request. A crawler that ignores robots.txt has to be blocked at your server or CDN instead.
What about blocking by IP or user agent at the firewall?+
That works where robots.txt is only a request — Cloudflare and most CDNs offer an AI-crawler toggle that blocks at the edge. Just be aware it is all-or-nothing in most implementations: it will block the search and user-fetch crawlers alongside the training ones, which is usually more than people intend.
Does this tool check every page, or just the domain?+
robots.txt is a single file at the root of a domain, so the rules it reports apply to the whole site. Whether a specific page is reachable and readable is a different question — our free page checker runs 31 checks on any URL, including whether that page is indexable.
This list is missing a crawler I have seen in my logs.+
Probably deliberately. We report the agents whose operators document them publicly, so we can say accurately what blocking one costs you. If a name shows up in your robots.txt that we do not recognise, the tool lists it separately rather than guessing what it does.
Letting them in is the floor, not the goal.
A crawler that can reach your page still has to find it worth quoting. The free page checker runs 31 checks on any URL to tell you whether it is.