Rightshold™ Sign up

Answers · AI crawlers

Do AI crawlers like GPTBot, ClaudeBot and PerplexityBot run JavaScript, or do they miss content and links that my site renders client-side?

Last updated

TL;DR: In every published measurement so far, most of them don't run it. Vercel and MERJ's study of crawler traffic, published December 17, 2024, found that OpenAI's crawlers (GPTBot, OAI-SearchBot and ChatGPT-User), Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot didn't render JavaScript. GPTBot and ClaudeBot did download JavaScript files, but as text, without executing them. The exceptions were Google, whose Gemini uses Googlebot's rendering infrastructure, and Applebot, which renders in a browser. Common Crawl's own FAQ says CCBot doesn't execute JavaScript either. A 41-day controlled test published by Search Engine Land on August 19, 2026 got the same answer for links: only Google's crawlers followed links injected by JavaScript, while GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, Meta-ExternalAgent, Amazonbot and PerplexityBot reached none of them; Bingbot reached three, apparently by another route. So if your text, prices or navigation only appear after a script runs, most AI crawlers and assistant fetches see an empty shell. None of these vendors documents rendering one way or the other, so treat this as measured behaviour that could change, put what matters in the server-rendered HTML, and test your own site.

What should I check, step by step?

1. Look at what a non-rendering crawler gets: curl your key pages (or view source with JavaScript off) and check that the main text, title, prices, product details and navigation links are in the HTML response itself.

  1. Server-render or pre-render anything you want AI crawlers and assistant fetches to read (SSR, static generation or incremental regeneration). Client-side rendering is fine for counters, chat widgets and other extras.
  2. Put internal links in plain <a href> tags in the HTML. Links that only exist after hydration can cut whole sections off for crawlers that don't run JavaScript.
  3. Keep an up-to-date sitemap, so crawlers can find deep pages even when a link is missing from the HTML.
  4. Don't put a JavaScript-only challenge or interstitial in front of crawlers you want. A crawler that doesn't run scripts can't pass it, so it sees the challenge page instead of your content.
  5. Don't block the JavaScript, CSS or API calls your pages need in robots.txt. Google and Apple render pages, and Apple says blocked resources can stop Applebot rendering the page properly.
  6. Test with a canary: inject one link with JavaScript to a unique URL that appears nowhere in your HTML or sitemap, then watch which user agents request it. Verify the requester's IP against the operator's published ranges or reverse DNS before you trust the user agent.
  7. Don't count .js requests from GPTBot or ClaudeBot as rendering. In Vercel's data they fetched script files without executing them.
  8. Re-check after big front-end changes and every few months; crawler behaviour isn't documented and can change without notice.

Do GPTBot and ClaudeBot execute JavaScript?

Vercel and MERJ's study (December 17, 2024) used traffic on nextjs.org and Vercel's network, checked against two job sites on different stacks. In one month it counted 569 million GPTBot fetches, 370 million from Claude, 314 million from Applebot and 24.4 million from PerplexityBot, against 4.5 billion from Googlebot.

Which AI crawlers render JavaScript?

Google documents that Googlebot crawls, then queues pages for rendering with an evergreen version of Chromium, then indexes the rendered page; rendering can be delayed while a page waits in the queue. Vercel's study found Gemini uses that same infrastructure.

How fast do AI crawlers find pages after JavaScript links are moved into HTML?

After 27 days, Stanula converted the JavaScript links to plain HTML links without changing anything else. GPTBot swept the formerly hidden sections within two days and picked up 250 pages it had never crawled; Bingbot added 212 and Meta-ExternalAgent followed by day 41.

How do I ask Rightshold about this (free, no key)?

Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Before I blame client-side rendering for AI assistants missing our product pages: what does example.com's robots.txt tell GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot for the /products/ path? Are any of them disallowed?”

One-click setup

{
  "tool": "read_ai_policy",
  "arguments": {
    "domain": "example.com",
    "path": "/products/"
  }
}

Rightshold reads a site's public crawl and AI-use signals and checks agent signatures you send it. It doesn't render pages, crawl your site or see your traffic. The crawler behaviour above is from the cited studies for the dates given; vendors don't document whether their crawlers render JavaScript, so results on your site may differ and may change.

What are the sources?

Related

Not legal advice. JSON · Markdown · All answers

Get a free API key in one call

No card and no email needed. 100 free credits; the license and policy checks stay free without a key.

curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_...   # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
  -H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'
Get a keyQuickstart