CoreCited
Guide

Claude SEO: how Claude searches the web, what it cites, and what its crawlers do

How Claude web search works, how it cites sources, what ClaudeBot, Claude-SearchBot and Claude-User do, and how to get your site cited by Claude.

Updated 12 min read12 sections
On this page · 12 sections

Claude SEO means making your site easy for Anthropic’s Claude to find, cite and describe correctly when it searches the web. Claude has searched the web on every plan since May 2025, cites a source in every search answer, and uses three separate crawlers, only one of which is about training.

Claude is a smaller source of traffic than ChatGPT, but its share of AI traffic has been growing. This guide covers how Claude web search works, how it cites, what its crawlers do and what blocking each one costs, and how to become a source Claude cites. Where Anthropic has said something, it is quoted; where something is only reported, it says so.

Key takeaways
1Claude has searched the web since March 2025, and on all plans worldwide since May 2025. Every search answer includes citations.
2Anthropic does not name its search provider; Brave Search is widely reported. Image results are powered by Bing.
3ClaudeBot is for training. Claude-SearchBot and Claude-User serve search; Anthropic says blocking them may reduce your visibility in Claude.
4To opt out of training but stay in Claude's answers, block ClaudeBot only.
5Claude sends far less traffic than ChatGPT, but its share of AI traffic has been growing.

What Claude SEO means

Definition · Claude SEO
The practice of making a website accessible to Anthropic’s search crawlers and useful enough that Claude cites it when it searches the web to answer a question.
~9%
Claude's share of generative AI web traffic, May 2026
Similarweb (third-party)
2.3%
of AI referrals to sites came from Claude, Q1 2026
BrightEdge (third-party)
3
Anthropic crawlers, each with a different job
Anthropic
$10
per 1,000 searches for developers using Claude's API tool
Anthropic

Sources: Similarweb[10], BrightEdge[11], Anthropic[3][4]. Anthropic does not publish a user count for the Claude apps.

How Claude got web search

Claude and the web
Mar 2025
Web search launches
In feature preview for paid Claude users in the United States, with free users and more countries 'coming soon'.
May 2025
Search for developers
Anthropic's API gains a web search tool, priced at $10 per 1,000 searches plus tokens.
May 2025All plans
Every plan, worldwide
Web search becomes available globally on all Claude plans, including free.
Feb 2026
Three crawlers, documented separately
Anthropic's crawler documentation splits into ClaudeBot, Claude-SearchBot and Claude-User, each with its own robots.txt consequence.
Sep 2026Latest
Current state
Search and web fetch count towards usage limits; team and enterprise owners switch web search on for their workspace.
Sources: Anthropic; Search Engine Journal (February 2026).

How Claude web search works

When a question needs current information, Claude searches the web, reads the results and writes an answer with citations[2]. Claude can also fetch a specific page a user gives it, pulling the whole article into its context[2].

From a question to a cited Claude answer
Based on Anthropic's help pages and developer documentation.
Decide to search
Claude decides a question needs the web, or the user turns web search on. On Team and Enterprise, an owner must enable it for the workspace.
Step 1: Decide to search. Claude decides a question needs the web, or the user turns web search on. On Team and Enterprise, an owner must enable it for the workspace.
Step 2: Search. Claude sends one or more searches. Anthropic does not name the provider; Brave Search is widely reported. Image results come from Bing.
Step 3: Read. Claude reads the results, and may fetch full pages, which is where Claude-User and your robots.txt come in.
Step 4: Answer. Claude writes a response grounded in what it read.
Step 5: Cite. Every web search response includes citations: links to sources and, where useful, quotes.
Why the search provider matters
Brave was added to Anthropic’s subprocessor list the day before web search launched, and early tests matched Claude’s citations to Brave’s results[6]. Anthropic has not confirmed it in its help pages. If it is Brave, being findable in Brave matters, and Brave says its crawler does not use a separate user agent and will not crawl anything Googlebot is blocked from[7]. So a page hidden from Google is likely hidden from Claude’s search as well.

Claude citations: how Claude cites sources

Anthropic’s help page is direct: “Every response includes citations, so you can easily verify sources yourself”, with direct citations, source links and relevant quotes[2]. For developers using the API, citations are always on for web search; each carries the page URL, title and up to 150 characters of the cited text, and Anthropic requires citations to be shown when API output is displayed to end users[4].

That 150-character limit is a useful discipline for writers. A citation can only quote a short passage, so the sentence you want quoted has to stand on its own: a clear claim, with its number or its name, in one sentence.

ClaudeBot, Claude-SearchBot and Claude-User

Anthropic uses three crawlers, and its documentation, updated in April 2026, gives each its own consequence if you block it[3]. Until early 2026 the documentation talked mainly about ClaudeBot[8], which is why many robots.txt files still treat “Claude” as one thing. Use the planner to see what each choice means:

Claude crawler planner
Choose what each Anthropic crawler may do. The effects are quoted from Anthropic's crawler documentation.
ClaudeBotTraining
Claude-SearchBotSearch indexing
Claude-UserFetches a page when a user asks
Add to your robots.txt
# All three Anthropic crawlers are allowed.
# Nothing needs to be added for them.
Anthropic says to repeat the rules on every subdomain you want to opt out, and that blocking its IP addresses may not reliably opt you out, because it stops Anthropic reading your robots.txt.
Myth
Blocking ClaudeBot removes you from Claude's answers.
What is true
ClaudeBot is for training. Anthropic says blocking it excludes future content from training, not from search.
Myth
Claude's user-triggered bot ignores robots.txt, like some others.
What is true
Anthropic says its bots honour robots.txt and makes no exception for Claude-User, unlike OpenAI's ChatGPT-User and Perplexity-User.
Myth
Blocking Anthropic's IP ranges is the safest opt-out.
What is true
Anthropic says IP blocking may not reliably opt you out, because it stops Anthropic reading your robots.txt.

What is ClaudeBot, and should you block it?

ClaudeBot is the crawler that turns up most in server logs, which is why “ClaudeBot” is searched far more than “Claude SEO”. It collects web content that could contribute to training Anthropic’s models, and it supports Crawl-delay if you want it to slow down[3].

Whether to block it is a content decision, not a visibility one. In August 2025 Cloudflare measured Anthropic’s crawl-to-refer ratio at nearly 50,000 crawls for every visit referred back, against 887 for OpenAI and 118 for Perplexity[9]. Publishers who see their content as the product often block training crawlers for that reason. For most businesses, whose content exists to be found, the cost of blocking ClaudeBot is low and the benefit is small too. What matters for visibility is keeping Claude-SearchBot and Claude-User open.

For the full list of AI crawlers and what each costs to block, see AI traffic analytics. To write the rules, use the robots.txt generator and tester, which has a “Block AI training only” preset.

How to get cited by Claude

Nobody outside Anthropic knows exactly how Claude chooses what to cite. These steps follow from how its search works and what it does with pages. Your ticks are saved in this browser.

Claude visibility checklist
0/7

Claude vs ChatGPT vs Perplexity for SEO

How the three answer engines differ
ClaudeChatGPTPerplexity
Search sourceNot named by Anthropic; Brave reportedSearch providers, including MicrosoftIts own crawler and index
Training crawlerClaudeBotGPTBotNone (says PerplexityBot is not for training)
Search crawlerClaude-SearchBotOAI-SearchBotPerplexityBot
User-triggered fetch honours robots.txt✓YesMay not applyGenerally ignores
Citations in every search answer✓Yes✓Yes✓Yes
Web search on the free plan✓Yes✓Yes✓Yes

Visibility does not transfer between them automatically. A brand can be cited by Perplexity and absent from Claude, which is why each engine is worth checking on its own. See the guides to ChatGPT SEO and Google AI Mode for the other two big surfaces.

Measuring traffic from Claude

Claude sends real but modest traffic. Across 74,752 sites using Ahrefs’ analytics in March 2026, AI assistants sent about 3.5 million visits, and this is how they split[12]:

Visits sent to websites by AI assistant, March 2026 (thousands)
Across 74,752 sites using Ahrefs Web Analytics. Approximate figures as reported.
Visits sent to websites by AI assistant, March 2026 (thousands)
ChatGPT2700
Perplexity230
Gemini230
Claude102
Source: Ahrefs, AI chatbot traffic, March 2026. Third-party data; some AI visits land in 'direct' and are not counted.

The trend matters more than the size. Similarweb reported Claude’s share of generative AI web traffic rising from about 2% in June 2025 to close to 9% in May 2026[10], and BrightEdge put Claude at 2.3% of AI referrals to sites in the first quarter of 2026[11].

In Google Analytics 4, visits from claude.ai arrive as referrals. GA4’s AI Assistant channel, added in May 2026, lists assistants by referrer, but its channel definition does not name Claude[13], so check where claude.ai sessions land in your own reports. Our guide to measuring AI traffic in GA4 covers the setup.

Using Claude for SEO work

“Claude for SEO” also means the other direction: using Claude AI as an SEO assistant. With web search on, it can research a topic, compare what the pages ranking for a query cover, draft a brief or a first draft, suggest headings and FAQ questions, and write structured data from a page you paste in. Two cautions apply to every AI tool, Claude included: check any statistic or quote it gives you against the source it cites, and have someone who knows the subject edit anything you publish. Our AI SEO guide covers which SEO tasks AI helps with and where it needs a person.

Questions people ask

What is Claude SEO?

Claude SEO is optimising a website so that Anthropic's Claude can find it, cite it and describe it accurately when it searches the web to answer a question. It combines crawler access (letting Anthropic's search and user-fetch crawlers in), clear, answer-first content, and presence in the sources Claude's search draws on.

Does Claude search the web?

Yes. Anthropic added web search to Claude on 20 March 2025 for paid users in the United States, and made it available globally on all Claude plans, including free, on 27 May 2025. On Team and Enterprise plans an owner must switch it on first. When it searches, every response includes citations.

What search engine does Claude use?

Anthropic's own help pages do not name one. Brave Search was added to Anthropic's subprocessor list the day before web search launched, and testing showed Claude's citations matching Brave's results, so Brave is widely reported as the provider. Anthropic's help page does say image results are powered by Bing.

What is ClaudeBot?

ClaudeBot is Anthropic's web crawler for collecting content that could contribute to training its models. Blocking it in robots.txt signals that your site's future content should be excluded from training. Anthropic runs two other crawlers: Claude-SearchBot for search indexing and Claude-User for fetching pages when a user asks.

Should I block ClaudeBot?

Only if you want to keep your content out of Anthropic's training data. Anthropic does not say blocking ClaudeBot affects search visibility. Blocking Claude-SearchBot or Claude-User is different: Anthropic says that may reduce your visibility in Claude's answers. To opt out of training but stay in search, block ClaudeBot alone.

Does Claude respect robots.txt?

Anthropic says all three of its crawlers honour robots.txt and support the Crawl-delay extension, and it makes no exception for its user-triggered Claude-User, unlike OpenAI and Perplexity for theirs. Rules have to be set on every subdomain you want to opt out, and Anthropic warns that blocking its IP addresses may not reliably work.

How does Claude cite sources?

In the Claude apps, every web search response includes citations, with direct links to sources and relevant quotes where appropriate. Through the API, citations are always on for web search, and each one carries the source URL, title and up to 150 characters of the cited text.

How do I get my website cited by Claude?

Allow Claude-SearchBot and Claude-User in robots.txt, make sure your pages are indexable by mainstream search engines, answer specific questions directly near the top of each section, include evidence Claude can quote, and earn mentions in the third-party sources that rank for your topic. Then check the questions that matter repeatedly to see whether Claude names you.

What is the difference between Claude-User and Claude-SearchBot?

Claude-SearchBot crawls to improve search results for Claude users, building what Claude knows about pages ahead of time. Claude-User visits a page at the moment a user's question needs it. Blocking either may reduce your visibility in Claude, in different ways.

How can I see traffic from Claude?

Visits from claude.ai show up as referrals in your analytics. Google Analytics 4 added an AI Assistant channel in May 2026; its release note lists Claude among the examples, but its channel definition does not name Claude, so check where claude.ai sessions land in your own reports. Visits from mobile apps may arrive without a referrer and appear as direct.

Keep reading

Sources

Anthropic’s blog, help centre and developer documentation are primary sources. Brave’s role is reported by Simon Willison from Anthropic’s subprocessor list. Cloudflare, Similarweb, BrightEdge and Ahrefs are third-party data with their own methods.

[1]Claude can now search the web — Anthropic, 20 March 2025, updated 27 May 2025
[2]Enable and use web search — Claude Help Center, updated 22 September 2026
[4]Web search tool — Claude Developer Platform, read 26 September 2026
[5]Web fetch tool — Claude Developer Platform, read 26 September 2026
[6]Anthropic's new web search uses Brave — Simon Willison, 21 March 2025
[7]Brave Search crawler — Brave, read 26 September 2026
[8]Anthropic's Claude bots make robots.txt decisions more granular — Search Engine Journal, 25 February 2026
[9]AI crawler traffic by purpose and industry — Cloudflare, 28 August 2025
[10]Generative AI statistics — Similarweb, 29 July 2026
[12]AI chatbot traffic — Ahrefs, March 2026
[13][GA4] Default channel group — Google Analytics Help, read 26 September 2026

Find out whether AI names you

Run one question through real AI engines and see the answer they give, who gets named in it, and whether you are in there at all. No account, no card.