{ "question": "Do AI crawlers like GPTBot, ClaudeBot and PerplexityBot run JavaScript, or do they miss content and links that my site renders client-side?", "slug": "do-ai-crawlers-like-gptbot-and-claudebot-run-javascript-or-see-client-side-rendered-pages", "url": "https://www.rightshold.com/answers/do-ai-crawlers-like-gptbot-and-claudebot-run-javascript-or-see-client-side-rendered-pages", "short_answer": "In every published measurement so far, most of them don't run it. Vercel and MERJ's study of crawler traffic, published December 17, 2024, found that OpenAI's crawlers (GPTBot, OAI-SearchBot and ChatGPT-User), Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot didn't render JavaScript. GPTBot and ClaudeBot did download JavaScript files, but as text, without executing them. The exceptions were Google, whose Gemini uses Googlebot's rendering infrastructure, and Applebot, which renders in a browser. Common Crawl's own FAQ says CCBot doesn't execute JavaScript either. A 41-day controlled test published by Search Engine Land on August 19, 2026 got the same answer for links: only Google's crawlers followed links injected by JavaScript, while GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, Meta-ExternalAgent, Amazonbot and PerplexityBot reached none of them; Bingbot reached three, apparently by another route. So if your text, prices or navigation only appear after a script runs, most AI crawlers and assistant fetches see an empty shell. None of these vendors documents rendering one way or the other, so treat this as measured behaviour that could change, put what matters in the server-rendered HTML, and test your own site.", "checklist": [ "Look at what a non-rendering crawler gets: curl your key pages (or view source with JavaScript off) and check that the main text, title, prices, product details and navigation links are in the HTML response itself.", "Server-render or pre-render anything you want AI crawlers and assistant fetches to read (SSR, static generation or incremental regeneration). Client-side rendering is fine for counters, chat widgets and other extras.", "Put internal links in plain tags in the HTML. Links that only exist after hydration can cut whole sections off for crawlers that don't run JavaScript.", "Keep an up-to-date sitemap, so crawlers can find deep pages even when a link is missing from the HTML.", "Don't put a JavaScript-only challenge or interstitial in front of crawlers you want. A crawler that doesn't run scripts can't pass it, so it sees the challenge page instead of your content.", "Don't block the JavaScript, CSS or API calls your pages need in robots.txt. Google and Apple render pages, and Apple says blocked resources can stop Applebot rendering the page properly.", "Test with a canary: inject one link with JavaScript to a unique URL that appears nowhere in your HTML or sitemap, then watch which user agents request it. Verify the requester's IP against the operator's published ranges or reverse DNS before you trust the user agent.", "Don't count .js requests from GPTBot or ClaudeBot as rendering. In Vercel's data they fetched script files without executing them.", "Re-check after big front-end changes and every few months; crawler behaviour isn't documented and can change without notice." ], "sections": [ { "heading": "What the crawler studies found", "points": [ "Vercel and MERJ's study (December 17, 2024) used traffic on nextjs.org and Vercel's network, checked against two job sites on different stacks. In one month it counted 569 million GPTBot fetches, 370 million from Claude, 314 million from Applebot and 24.4 million from PerplexityBot, against 4.5 billion from Googlebot.", "It found no JavaScript rendering by OpenAI's three crawlers, ClaudeBot, Meta-ExternalAgent, Bytespider or PerplexityBot. JavaScript files were 11.50% of ChatGPT's fetches and 23.84% of Claude's, but the study found the files weren't executed. It noted that content already in the initial HTML, such as JSON data or React Server Components payloads, may still be read. Microsoft Copilot was left out because it has no unique user agent.", "Vinicius Stanula's experiment for Search Engine Land (August 19, 2026) split one site's 21 sections in two: in 11 every child link was in the HTML, in 10 every child link was added by JavaScript, with sitemaps and breadcrumbs removed. GPTBot crawled all 748 HTML-linked hierarchy pages on day one and none of the 293 JavaScript-linked ones; ClaudeBot, Meta-ExternalAgent and Amazonbot showed the same split. GoogleOther reached its first JavaScript-linked page 54 minutes after launch and Googlebot after 9.9 hours, though Googlebot reached only 2% of those pages in 27 days. The author notes it was one new domain with no external links." ] }, { "heading": "The crawlers that do render", "points": [ "Google documents that Googlebot crawls, then queues pages for rendering with an evergreen version of Chromium, then indexes the rendered page; rendering can be delayed while a page waits in the queue. Vercel's study found Gemini uses that same infrastructure.", "Apple says Applebot may render a website's content within a browser, and that if JavaScript, CSS and other resources are blocked in robots.txt it may not be able to render the page properly.", "Common Crawl's FAQ says CCBot doesn't execute JavaScript or use cookies, so the many datasets built from Common Crawl only contain what was in the HTML." ] }, { "heading": "What changed when the links moved into HTML", "points": [ "After 27 days, Stanula converted the JavaScript links to plain HTML links without changing anything else. GPTBot swept the formerly hidden sections within two days and picked up 250 pages it had never crawled; Bingbot added 212 and Meta-ExternalAgent followed by day 41.", "ClaudeBot kept recrawling pages it already knew but hadn't picked up any of the newly linked pages by day 41, and Amazonbot had already left the site. Googlebot had visited one page in the converted sections by day 41. Getting links into the HTML before launch avoids waiting on each crawler's next pass.", "ChatGPT-User, which fetches pages when someone asks ChatGPT a question, reached dozens of HTML-linked pages during the test and none behind JavaScript links, so the gap affects live assistant answers as well as training crawls." ] } ], "checks": [], "ask_rightshold": { "start": "https://www.rightshold.com/start?ref=seo", "mcp_endpoint": "https://www.rightshold.com/mcp", "ask": "Before I blame client-side rendering for AI assistants missing our product pages: what does example.com's robots.txt tell GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot for the /products/ path? Are any of them disallowed?", "mcp_call": { "tool": "read_ai_policy", "arguments": { "domain": "example.com", "path": "/products/" } }, "also": [ "Free, no key, cached 6 hours. It reads robots.txt for 14 AI crawlers and user-fetch agents (training, search and user-fetch tokens), ai.txt and W3C TDMRep, and whether llms.txt exists, then summarizes what each bot is told for the path you give.", "It rules out one cause: a robots.txt block. It doesn't fetch or render your pages, test JavaScript, read your logs or see what a crawler actually indexed." ], "scope": "Rightshold reads a site's public crawl and AI-use signals and checks agent signatures you send it. It doesn't render pages, crawl your site or see your traffic. The crawler behaviour above is from the cited studies for the dates given; vendors don't document whether their crawlers render JavaScript, so results on your site may differ and may change." }, "sources": [ { "title": "Vercel and MERJ: The rise of the AI crawler (December 17, 2024)", "url": "https://vercel.com/blog/the-rise-of-the-ai-crawler" }, { "title": "Search Engine Land: JavaScript links can make your pages invisible to AI search (Vinicius Stanula, August 19, 2026)", "url": "https://searchengineland.com/javascript-links-pages-invisible-ai-search-485228" }, { "title": "Google Search Central: Understand JavaScript SEO basics", "url": "https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics" }, { "title": "Apple Support: About Applebot (rendering and robots rules)", "url": "https://support.apple.com/en-us/119829" }, { "title": "Common Crawl: FAQ (JavaScript is not executed)", "url": "https://commoncrawl.org/faq" } ], "related": [ { "title": "How to make a website readable to AI agents with llms.txt and markdown pages", "url": "https://www.rightshold.com/answers/how-to-make-a-website-readable-to-ai-agents-with-llms-txt-and-markdown" }, { "title": "How to tell ChatGPT-User fetches from AI training crawlers in server logs", "url": "https://www.rightshold.com/answers/how-to-tell-chatgpt-user-fetches-from-ai-training-crawlers-in-server-logs" }, { "title": "Does Anubis proof-of-work stop AI scrapers, and what does it cost real visitors?", "url": "https://www.rightshold.com/answers/does-anubis-proof-of-work-stop-ai-scrapers-and-what-does-it-cost-real-visitors" }, { "title": "Why Google Analytics shows almost no AI bot traffic, and how to measure it", "url": "https://www.rightshold.com/answers/why-google-analytics-shows-almost-no-ai-bot-traffic-and-how-to-measure-it" } ], "computed_from": "Curated guidance; license verdicts computed from Rightshold's license profiles at render time. Not legal advice.", "updated": "2026-11-10" }