Rightshold™ Sign up

Answers · AI crawlers

Does robots.txt stop AI crawlers, and how do I block AI training without leaving AI search?

Last updated

TL;DR: robots.txt is a request, not a lock. RFC 9309 says its rules 'are not a form of access authorization', but the big AI operators say their automatic crawlers follow it, and most use separate tokens for training and for search, so one file can say no to training and yes to AI search. Disallow GPTBot (OpenAI training), ClaudeBot (Anthropic training), Google-Extended (Gemini training and grounding; Google says it doesn't affect Google Search), Applebot-Extended (Apple's foundation models; it doesn't crawl, so Applebot search continues) and CCBot (Common Crawl). Leave OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Applebot allowed. Fetches a person asks an assistant to make follow different rules: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Anthropic says disallowing Claude-User stops those fetches. For scrapers that ignore robots.txt or borrow a known bot's name, robots.txt does nothing; that takes a firewall rule checked against the operators' published IP lists.

What should I check, step by step?

1. Write one group per training token: User-agent: GPTBot, User-agent: ClaudeBot, User-agent: Google-Extended, User-agent: Applebot-Extended and User-agent: CCBot, each followed by Disallow: /. Several User-agent lines can share one group.

  1. Name the search crawlers you want in their own group with Allow: / (OAI-SearchBot, Claude-SearchBot, PerplexityBot). RFC 9309 says a crawler obeys the group that names it and uses the * group only when no group matches, so a strict * group won't block them once they have their own.
  2. Don't block Googlebot or Applebot unless you want to leave those search engines. Google says Google-Extended doesn't affect inclusion or ranking in Google Search, and Apple says pages that disallow Applebot-Extended can still be included in its search results.
  3. Decide about user-triggered fetchers on purpose. Disallowing Claude-User stops Claude's user-requested fetches. For ChatGPT-User and Perplexity-User, robots.txt may not hold, so block them at the firewall if you must, knowing assistant users then can't read or cite you.
  4. Put the file at the root of every host. Anthropic says to do this for each subdomain you want opted out, and robots.txt on www doesn't cover blog.example.com.
  5. Keep robots.txt reachable. Under RFC 9309, a 4xx lets crawlers access anything, and a 5xx means crawlers must assume everything is disallowed. Crawlers should re-read the file at least every 24 hours, and OpenAI and Perplexity say changes can take about 24 hours to apply.
  6. Don't use IP blocks to opt out of training. Anthropic says blocking its IPs may not work reliably as an opt-out, because then it can't read your robots.txt.
  7. For anything that ignores the file, enforce at the edge: verify a claimed bot by its operator's IP list or reverse DNS (OpenAI, Perplexity, Apple and Common Crawl publish these), then block or rate limit what fails.

What robots.txt can and can't do

RFC 9309 standardized the Robots Exclusion Protocol in 2022. It describes rules that crawlers are requested to honor and says they are not access authorization. Its security section adds that listing paths publicly makes them easier to find, so robots.txt is the wrong place to hide private pages.

The tokens that only control use

Google-Extended and Applebot-Extended don't crawl. Google crawls with its existing user agents and reads Google-Extended only as a control token for Gemini training and grounding. Apple reads Applebot-Extended only to decide how data Applebot crawled may be used. You won't see either in your logs.

How do I ask Rightshold about this (free, no key)?

Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “What does example.com's robots.txt tell GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and the AI search and user-fetch bots? Is training opted out while search stays allowed?”

One-click setup

{
  "tool": "read_ai_policy",
  "arguments": {
    "domain": "example.com"
  }
}

Rightshold reads a site's public robots.txt and AI-use signals on request. It does not enforce robots.txt, block crawlers, see your traffic or tell you whether an operator actually honored your rules, and these signals are not a license. Not legal advice.

What are the sources?

Related

Not legal advice. JSON · Markdown · All answers

Get a free API key in one call

No card and no email needed. 100 free credits; the license and policy checks stay free without a key.

curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_...   # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
  -H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'
Get a keyQuickstart