Next.js SEO is mostly solved by the framework’s defaults: pages render on the server, metadata has a proper API, and sitemaps and robots files are built in. The gaps appear with AI crawlers, which do not run JavaScript, and with one newer behaviour, streaming metadata, that puts your title and description somewhere they may not look. This site runs on Next.js, so everything below was checked on a real deployment.
What crawlers receive from a Next.js page
In the App Router, “layouts and pages are Server Components” by default[4]. Even components marked 'use client' are prerendered to HTML on the first load, which Next.js uses “to immediately show a fast non-interactive preview of the route”. So the common fear that Next.js hides content from crawlers is mostly wrong. The exception is content fetched in the browser after load, in a useEffect or a client-side data library: that never reaches the server HTML.
That exception matters more now. Google renders JavaScript, but its own guidance says server rendering “is still a great idea” because “not all bots can run JavaScript”[9]. Vercel measured which ones: the major AI crawlers, from OpenAI, Anthropic, Meta, ByteDance and Perplexity, do not render JavaScript, while Gemini (through Googlebot) and AppleBot do[8].
| Server-rendered content | Content loaded by client JavaScript | |
|---|---|---|
| Googlebot | Yes | Yes, after rendering |
| GPTBot, OAI-SearchBot | Yes | No |
| ClaudeBot | Yes | No |
| PerplexityBot | Yes | No |
The streaming metadata catch
Next.js 15.2 introduced streaming for generateMetadata. When metadata is resolved at request time, Next.js can send the page without waiting and “the resulting metadata tags are appended to the <body> tag”. Next.js says it verified this works for bots that run JavaScript, such as Googlebot. For “HTML-limited bots” it keeps the old behaviour and puts metadata in the head, detected by user agent[1].
The default list of HTML-limited bots covers link previewers and several search engines, such as facebookexternalhit, Bingbot and Twitterbot. It does not include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot or PerplexityBot[3]. We checked what that means in practice on this site, which runs Next.js 15.5 on Vercel. Its dynamic /unsubscribe page served the <title> inside <body> to GPTBot, ClaudeBot, Googlebot and Chrome, and inside <head> to facebookexternalhit. Try a user agent:
Fixing it
The best fix is to not need it: prerender. If generateMetadata only uses route params and build-time data, and the route is listed in generateStaticParams, the metadata is in the initial HTML for every crawler. Note that you “must always return an array” from generateStaticParams, even an empty one, or the route renders dynamically[5].
Where pages must render per request, set htmlLimitedBots. Your value replaces the default list[2], so keep the defaults and add the AI crawlers:
// next.config.ts — the default list plus AI crawlers.
// Setting htmlLimitedBots replaces the default, so the default is included.
const nextConfig = {
htmlLimitedBots: /[\w-]+-Google|Google-[\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti|googleweblight|GPTBot|OAI\-SearchBot|ChatGPT\-User|ClaudeBot|Claude\-User|Claude\-SearchBot|PerplexityBot|Perplexity\-User|CCBot/i,
};
export default nextConfig;Next.js also documents htmlLimitedBots: /.*/ to switch streaming off entirely, with the warning that it “could lead to longer response times”[1].
The rest of Next.js SEO, briefly
- Metadata. Export
metadataorgenerateMetadatafrom each page, with a unique title, a description andalternates.canonical. It only works in Server Components, because metadata “must be resolved on the server”[1]. - Sitemap.
app/sitemap.tsgenerates the XML and is cached by default; split large sites withgenerateSitemaps[7]. - robots.txt.
app/robots.tsaccepts separate rules per user agent, which is how you treat GPTBot and OAI-SearchBot differently[6]. The robots.txt generator writes the rules. - Share images. An
opengraph-image.tsxnext to a page generates its card at build time. - Structured data. Render JSON-LD in a server component so it is in the HTML. The schema generator builds it.
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \ https://example.com/page | grep -o "<title>.*</title>\|</head>"
If the title appears after </head>, metadata is being streamed. If nothing prints at all, the page may be redirecting, and one that redirects in circles is covered in fixing ERR_TOO_MANY_REDIRECTS. For which AI crawlers actually visit, see log file analysis for AI crawlers.
Questions people ask
Is Next.js good for SEO?
Yes, if pages are rendered on the server. In the App Router, pages and layouts are Server Components by default, so their content is in the HTML a crawler receives. The risks come from content that only appears after client-side JavaScript runs, and from metadata resolved at request time.
How do I do SEO in Next.js?
Export a metadata object or generateMetadata from each page for title, description, canonical and Open Graph; add app/sitemap.ts and app/robots.ts; prerender pages where you can with generateStaticParams; keep important content out of client-only components; and add an opengraph-image for share cards.
Can AI crawlers read Next.js sites?
They can read whatever is in the server HTML. Vercel found that the major AI crawlers, including OpenAI's and Anthropic's, do not execute JavaScript, so text that only appears after scripts run is invisible to them. Server-rendered and prerendered pages are fine.
What is streaming metadata in Next.js?
Since Next.js 15.2, when generateMetadata runs at request time, Next.js can send the page before the metadata is ready and append the tags to the body afterwards. Bots on its HTML-limited list get blocking metadata in the head instead. AI crawlers are not on the default list.
Do I need htmlLimitedBots?
Only if pages resolve metadata at request time and you want AI crawlers to get it in the head. Prerendered pages already have metadata in the head for everyone. If you set htmlLimitedBots, include the default list too, because your value replaces it.