How to find a sitemap of a website: robots.txt, the common paths, and every platform

The fastest way to find the sitemap of a website is to open its robots.txt file, at example.com/robots.txt, and look for a line that starts with Sitemap:. That is where the owner tells every crawler where the file lives, and it can point anywhere. Only if there is no such line do you start guessing paths, and the right guess depends on the platform. This guide gives the order to try things in, the paths each platform documents, and the real cases where a guess fails.

If you would rather not type URLs, the free sitemap validator works as a sitemap finder too. Give it a domain and it reads robots.txt, then tries /sitemap.xml and /sitemap_index.xml, then opens what it finds.

Sources: Bing’s study of sites with at least one URL in its index^[7]; the sitemap protocol^[1]; Google^[5]. The New York Times count is what its robots.txt served when we fetched it.

Method 1: read the robots.txt file

The sitemap protocol lets a site name its sitemap in robots.txt with one line, Sitemap: followed by the full URL. The line “is independent of the user-agent line, so it doesn’t matter where you place it in your file”, and a file may list more than one^[1]. Google reads it the same way: you can put the line “anywhere in your robots.txt file”, and “there’s no limit to the number of sitemaps you can include”^[2]. Bing names robots.txt as one of its two ways to submit a sitemap^[6]. The robots.txt standard itself leaves these lines outside the user-agent groups^[15].

That makes robots.txt the one place every crawler can look, and the first place you should. Open it in a browser, or run this:

# 1. Read the robots.txt and keep only the Sitemap lines
curl -s https://www.example.com/robots.txt | grep -i '^sitemap:'

# Windows PowerShell
(Invoke-WebRequest https://www.example.com/robots.txt).Content -split "`n" | Select-String '^sitemap:'

Here is what real robots.txt files said when we fetched them on 11 October 2026:

• Stripe declares https://stripe.com/sitemap/sitemap.xml. Its /sitemap.xml returned a 404. A path guess would have told you Stripe has no sitemap; robots.txt shows it has a sitemap index with nine child files.
• The New York Times lists 25 Sitemap: lines, including gzipped files such as /sitemaps/new/news.xml.gz and separate files for Wirecutter, The Athletic and Games. No single path would find all of them.
• Gymshark lists the same line, https://www.gymshark.com/sitemap.xml, three times. Harmless, but a sign the file is assembled by more than one setting.
• Ahrefs has no Sitemap: line at all, yet ahrefs.com/sitemap.xml answered 200 with a sitemap index of seven files. That is the case for method 2.

Method 2: try the common sitemap paths

Google recommends putting a sitemap at the site root, because a sitemap only covers the folder it sits in unless you submit it in Search Console^[2]. So most sitemaps live at the root, under a handful of names. Try them in this order, and look at the response: a 200 with an XML content type is a sitemap; a 200 with an HTML page is not.

# 2. No Sitemap line? Try the common paths and print the status and content type
for p in sitemap.xml sitemap_index.xml wp-sitemap.xml sitemap.txt sitemap.xml.gz; do
  printf '%-20s ' "$p"
  curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "https://www.example.com/$p"
done

The name is set by whatever generates the file, so the quickest route is to know the platform. Every path below comes from the platform’s own documentation.

Sources: WordPress^[8], Yoast^[9], Rank Math^[10], Shopify^[11], Wix^[12], Squarespace^[13], Next.js^[14].

A few details from those pages save time. WordPress core caps each file at 2,000 entries by default and has a filter that switches its sitemaps off, which SEO plugins can use^[8]. Yoast says plainly that if you see a sitemap at example.com/sitemap.xml, “your sitemap is not generated by Yoast SEO”^[9]. On the live site, Yoast’s own /wp-sitemap.xml answered with a 301 redirect to /sitemap_index.xml when we checked. Wix gives every site a sitemap index and updates it whenever you change the site^[12]. Squarespace says its updates “usually appear within an hour, though can take up to 24 hours”^[13].

Two variants are easy to miss. Sitemaps can be gzipped, so try .xml.gz as well as .xml^[1]. And the protocol accepts a plain text file with one URL per line^[1]; Google lets it have any name ending in .txt^[2].

Method 3: open the sitemap index

The file you find first is often not a list of pages. A sitemap may hold at most 50,000 URLs and 50 MB uncompressed, so larger sites split their URLs into several files and list those files in a sitemap index^[1]. You can tell them apart from the root element: <sitemapindex> lists sitemaps, <urlset> lists pages.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/post-sitemap.xml</loc>
    <lastmod>2026-10-09T07:32:54+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/page-sitemap.xml</loc>
  </sitemap>
</sitemapindex>

Open each child <loc> to reach the page URLs. On real sites the tree can be wide. When we ran the validator’s checks on TechCrunch on 11 October 2026, its robots.txt named two sitemaps, and the main index listed more than 2,000 child files. Yoast’s index listed 21 child sitemaps; the 20 we could read held 1,698 URL entries, 36 of them duplicates. That is why a sitemap finder that stops at the first file tells you little.

Method 4: Search Console and Bing Webmaster Tools

For a site you own, the search engines will tell you what they know. Search Console’s Sitemaps report shows the sitemaps submitted for the property, when each was last read, and its status^[3]. Two limits catch people out:

• It only lists submitted sitemaps. The report “does not show any sitemaps discovered through a robots.txt reference or other discovery methods”, even if Google uses them^[3]. An empty report does not mean Google has never seen your sitemap.
• It is per property. A sitemap submitted under https://example.com will not show under https://www.example.com. Google’s advice is to check you are not “confusing http/https, or www/non-www properties”^[3].

Bing Webmaster Tools has a Sitemaps section that shows the submission status, the last read date and processing errors, and Bing says it revisits a known sitemap “typically at least once per day”^[6]. The Bing Webmaster Tools guide covers setting it up.

Method 5: the last resorts

Look at the page footer. Some sites link a “Sitemap” page from the footer. That is usually an HTML sitemap, a page for people, not the XML file search engines read. Useful, but not the same thing.

Try a site: search, with low hopes. Searching site:example.com sitemap only finds a sitemap if Google indexed the file, and there is no reason it should. Yoast says “XML sitemaps do not need to be indexed by search engines in order to be read or parsed” and sends X-Robots-Tag: noindex, follow on every sitemap response^[9]; the header was there when we requested yoast.com’s own index. Google also warns that a site: search “doesn’t necessarily return all the URLs that are indexed”^[4].

Accept that there may be none. A sitemap is optional. Google says a site of “about 500 pages or fewer” that is well linked internally may not need one^[5], and Bing’s 2023 study found that 42% of hosts with a URL in its index had no XML sitemap Bing knew about^[7]. If robots.txt has no line and the paths return 404, that is your answer.

Once you have found it, check it

Finding the file is half the job. A sitemap can load and still be wrong: listing URLs that redirect, return 404, carry noindex or are blocked by robots.txt. Google asks you to list the canonical URLs you want in search results^[2], so each of those is an entry that should not be there. Three quick checks once you have the URL:

• Does it parse? The sitemap validator checks the XML, namespace, dates and limits, then requests a sample of the listed URLs.
• Are the dates real? When we checked www.wix.com on 11 October 2026, 79% of its 10,525 URLs shared one <lastmod> timestamp, and one date was in the future. The guide to lastmod explains why that matters.
• Can every crawler fetch what it lists? TechCrunch’s robots.txt disallowed GPTBot and ClaudeBot from all 10,023 URLs our check read, while Googlebot and Bingbot were allowed. That can be a deliberate choice. If it is not one on your site, the robots.txt tester shows which rule matches.

New to the format? What a sitemap is walks through every tag with annotated examples. And if Search Console says the file you found could not be read, that guide goes through each cause.

From one sitemap to every page

The validator samples URLs from the sitemap it finds. CoreCited’s GEO Audit starts from the same discovery, robots.txt and then the sitemap, and goes on to crawl the pages it lists. The free account audits 10 pages; every plan limit is on the pricing page.

Questions people ask

Sources