Robots.txt generator and tester
Create a robots.txt file in a few clicks, decide crawler by crawler which search engines and AI bots may read your site, then use the built-in robots.txt checker to test any URL and see exactly which line allows or blocks it.
# robots.txt generated by CoreCited — https://corecited.com/tools/robots-txt-generator # Everyone else User-agent: * Disallow:
A text file that tells crawlers where not to go.
A robots.txt file is a plain text file at the root of a website, such as example.com/robots.txt, that tells web crawlers which URLs they may request. It is defined by the Robots Exclusion Protocol, standardised as RFC 9309 in 2022, and every major search engine and AI company reads it before crawling.
It controls crawling, not indexing, and it is a request, not a lock. Well-behaved crawlers follow it. It does not hide a page from anyone who types the URL, and it does not guarantee a page stays out of search results. For that you need a noindex tag on a page that crawlers are allowed to read.
Every directive a robots.txt can hold, and what it does.
A file is a list of groups. Each group starts with one or more User-agent lines and holds the Allow and Disallow rules for those crawlers. Directive names are case-insensitive; the paths in them are not.
| Directive | Example | What it does |
|---|---|---|
| User-agent | User-agent: Googlebot | Starts a group and names who it is for. * means every crawler not named elsewhere. Several User-agent lines in a row share one group. |
| Disallow | Disallow: /private/ | A path the crawler should not fetch. An empty value (Disallow:) blocks nothing. |
| Allow | Allow: /private/public/ | Re-opens a path inside a disallowed one. The longer, more specific rule wins. |
| Sitemap | Sitemap: https://example.com/sitemap.xml | Where your XML sitemap is. Must be a full URL. It can go anywhere in the file and belongs to no group. |
| Crawl-delay | Crawl-delay: 5 | Seconds to wait between requests. Bing honours it; Google ignores it. |
| * | Disallow: /*.pdf | Matches any run of characters inside a path. |
| $ | Disallow: /*.pdf$ | Anchors the rule to the end of the URL, so /file.pdf?x=1 no longer matches. |
| # | # a comment | Everything after # on a line is ignored. |
The order of lines does not matter. Specificity does.
The most common robots.txt bug is assuming rules are read top to bottom, like a firewall. They are not. A crawler finds the one group meant for it, then picks the single most specific rule that matches the URL. These are the rules the tester above applies:
Disallow: /admin/ under User-agent: *, then add a separate group for GPTBot to allow it. GPTBot now ignores the * group entirely, including the admin rule. The generator above avoids this by repeating shared rules inside every named group that is not a full block.Eight robots.txt examples you can copy.
Each sample below is a complete file. Swap example.com for your domain, or paste it into the tester to see what it does to a URL before you upload it.
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
User-agent: * Disallow: /
User-agent: * Disallow: /admin/ Disallow: /cart/
User-agent: * Disallow: /*.pdf$ Disallow: /*.zip$
User-agent: * Disallow: /*?sort= Disallow: /*?sessionid=
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: / User-agent: * Disallow:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/wp-sitemap.xml
User-agent: * Disallow: /search Disallow: /share-widget Allow: / Sitemap: https://example.blogspot.com/sitemap.xml
Block AI training without disappearing from AI answers.
AI companies run separate crawlers for separate jobs. 7 of the crawlers below collect training data; blocking them costs no visibility today. 3 build the search indexes that ChatGPT, Claude and Perplexity cite from; blocking those removes you from their answers. Most sites that “block AI” only meant the first kind.
| User-agent | Operator | Job | What blocking it costs you |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Your content is not used to train future OpenAI models. It does NOT remove you from ChatGPT's answers — that is OAI-SearchBot. |
| OAI-SearchBot | OpenAI | AI search index | You are removed from ChatGPT's search index. This is the one that decides whether ChatGPT can cite you. |
| ChatGPT-User | OpenAI | Fetches on a user's request | Asks ChatGPT not to open your page when a user's question needs it. OpenAI says robots.txt rules "may not apply" to these user-initiated fetches, so this line may not stop it; blocking at your server or CDN will. |
| ClaudeBot | Anthropic | Model training | Your content is not used to train Claude. |
| Claude-SearchBot | Anthropic | AI search index | You are removed from the index Claude searches when answering. |
| Claude-User | Anthropic | Fetches on a user's request | Claude cannot fetch your page on a user's behalf. |
| PerplexityBot | Perplexity | AI search index | You are removed from Perplexity's index. Perplexity cites sources heavily, so this is usually the most costly single block. |
| Perplexity-User | Perplexity | Fetches on a user's request | Asks Perplexity not to open your page for a user's question. Perplexity says this fetcher generally ignores robots.txt rules, so block it at your server or CDN if you mean it. |
| Google-Extended | Model training | Your content is not used for Gemini training or grounding. It does NOT remove you from Google Search or AI Overviews — see the note below. | |
| Applebot-Extended | Apple | Model training | Your content is not used to train Apple Intelligence. |
| Meta-ExternalAgent | Meta | Model training | Your content is not used to train Meta's models. |
| Bytespider | ByteDance | Model training | Your content is not used to train ByteDance's models. |
| CCBot | Common Crawl | Model training | You leave the Common Crawl dataset, which many model builders train on — so this one block affects more than one company. |
Six robots.txt mistakes that quietly cost traffic.
Shipping the staging file
A Disallow: / written for a staging site and deployed to production is the most expensive single line in SEO. Check the live file after every launch.
Using it to hide a page from Google
A disallowed URL can still be indexed from links, with no snippet. To keep a page out of results, let it be crawled and add noindex.
Blocking CSS and JavaScript
Google renders pages like a browser. Blocking the files your layout needs can make it misread the page. Leave assets crawlable.
Assuming top-to-bottom order
Order is irrelevant. The longest matching rule wins, and a crawler with its own group ignores the * group entirely.
Listing secret URLs
The file is public. Disallow: /private-admin-2/ tells everyone where to look. Protect private areas with a login instead.
Blocking every AI bot by name
Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those assistants' answers. Block training crawlers only, unless that is really what you want.
How to add robots.txt to your site.
Google's old robots.txt tester is gone. Here is what replaced it.
Google Search Console used to include a robots.txt Tester. Google retired it in December 2023 and replaced it with the robots.txt report, which shows the robots.txt files Google found for your site, when it last fetched them and any problems it had reading them, and lets you ask Google to recrawl a file you have just fixed. It no longer lets you test a URL against the rules.
That is the gap the tester above fills: paste any file, test any URL, and see the deciding rule. To confirm what Google does with a specific live page, use the URL Inspection tool in Search Console, which reports whether the page is blocked by robots.txt.
Robots.txt questions, answered
How do I create a robots.txt file?+
Pick a starting point in the generator above, choose which crawlers to allow or block, add any folders to keep out and your sitemap URL, then copy or download the result. Save it as a plain text file named robots.txt and upload it to the root of your domain, so it is reachable at https://yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored.
How do I check or test my robots.txt?+
Paste the file into the tester above, enter a URL or path, and choose a crawler. It tells you whether that crawler may fetch that URL, and which line decided it. It follows RFC 9309 and Google's documented rules. To check the file Google has actually fetched from your live site, use the robots.txt report in Google Search Console. To see which AI crawlers your live file lets in, use our AI Crawler Checker.
What does 'Disallow: /' do?+
It blocks the whole site for every crawler the group applies to. Under 'User-agent: *' it tells every well-behaved crawler, Google included, to stay out entirely, which removes the site from search over time. It is right for a staging site and a mistake almost everywhere else. The opposite, 'Disallow:' with nothing after it, allows everything.
How do I disallow all crawlers?+
Use 'User-agent: *' followed by 'Disallow: /'. Choose the 'Block everything' preset above to generate it. Remember that robots.txt is a request, not a lock: well-behaved crawlers honour it, but it does not stop anyone determined to fetch your pages, and it does not hide URLs that other sites already link to.
Does robots.txt stop a page from appearing in Google?+
Not reliably. Disallow stops Google crawling a page, but a blocked URL can still be indexed without its content if other pages link to it. To keep a page out of search results, allow it to be crawled and add a noindex robots meta tag or X-Robots-Tag header. Google has to be able to crawl the page to see the noindex.
Should I add my sitemap to robots.txt?+
Yes, it costs nothing and helps crawlers other than Google find it. Add a line such as 'Sitemap: https://example.com/sitemap.xml' anywhere in the file. It must be a full URL, and you can list more than one. Sitemap lines are not tied to a user-agent group.
What is the difference between Allow and Disallow?+
Disallow closes a path and Allow opens one. When both match a URL, the rule with the longer path wins, because it is more specific. So 'Disallow: /shop' with 'Allow: /shop/sale' keeps crawlers out of /shop but lets them into /shop/sale. If an Allow and a Disallow of exactly the same length both match, Allow wins.
Do wildcards work in robots.txt?+
Yes. '*' matches any sequence of characters and '$' marks the end of the URL. 'Disallow: /*.pdf$' blocks every URL ending in .pdf, and 'Disallow: /*?sort=' blocks any URL with a sort parameter. Google and Bing support both. Paths are case-sensitive, so '/Admin' and '/admin' are different rules.
Should I block AI crawlers like GPTBot and ClaudeBot?+
It depends on what the crawler does. Training crawlers such as GPTBot and ClaudeBot collect content for future models, and blocking them costs you no visibility today. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether those assistants can cite you, and blocking them removes you from their answers. The 'Block AI training only' preset keeps you in AI answers while opting out of training.
Does Google respect Crawl-delay?+
No. Google ignores the Crawl-delay line. Bing honours it, which is why the generator writes it for Bingbot only. If Googlebot is putting too much load on your server, the fix is on the server side, for example by returning 429 or 503 responses temporarily.
How do I edit robots.txt in WordPress or Blogger?+
WordPress serves a virtual robots.txt until you replace it. Upload a real robots.txt file to your site's root, or edit it through your SEO plugin. In Blogger, go to Settings, then Crawlers and indexing, turn on 'Enable custom robots.txt', and paste the file. Pick WordPress or Blogger in the generator above to start from that platform's defaults.
Is robots.txt a security measure?+
No, and treating it as one backfires. The file is public, so listing '/secret-admin/' in it tells everyone where to look. Protect private areas with authentication. Use robots.txt only to manage what well-behaved crawlers spend their time on.
Letting crawlers in is step one. Being worth citing is step two.
A crawler that can reach your page still has to find it worth quoting. The free page checker runs 31 checks on any URL, and the visibility checker shows whether AI engines name you today.