{
  "question": "Why isn't an AI-training opt-out or robots.txt preference access control, and what should I use instead?",
  "slug": "why-an-ai-training-opt-out-or-robots-txt-is-not-access-control",
  "url": "https://www.rightshold.com/answers/why-an-ai-training-opt-out-or-robots-txt-is-not-access-control",
  "short_answer": "robots.txt, Google-Extended, Content-Signal: ai-train=no, ai.txt and TDM reservations are requests about how a well-behaved crawler should fetch or use public pages. They are not a lock. RFC 9309 says robots.txt rules \"are not a form of access authorization\", and Google says robots.txt is not a way to keep a page out of Google; use noindex or password protection. A scraper that ignores the file, or borrows GPTBot's user agent string, still gets every page your server hands to anonymous requests. If a page must not be read, put it behind authentication (session, SSO, signed URLs) or a CDN/WAF rule that blocks before the origin answers. Then verify crawler identity in your logs: check the claimed user agent against the operator's published IP list or a reverse-plus-forward DNS lookup, or check Web Bot Auth signatures. Keep training opt-outs for what they are good at: telling compliant AI operators how they may use pages you deliberately leave public.",
  "checklist": [
    "Sort your URLs into three groups: public and fine for AI training, public but not for training (an opt-out is enough here), and must-not-be-read (needs a lock, not a preference).",
    "Anything in the must-not-be-read group goes behind authentication: a login session, SSO, HTTP auth on staging, or short-lived signed URLs for files. Anonymous requests should get a 401, 403 or 404 instead of the content.",
    "Don't list secret paths in robots.txt. Anyone can read the file, so a Disallow: /internal-reports/ line tells scrapers where to look.",
    "Add CDN/WAF rules for crawler classes you don't want at all, for example Cloudflare's AI-bot blocking or a custom rule on path plus verified-bot status, so the request stops at the edge.",
    "Keep training opt-outs in robots.txt (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and, if you use them, Content-Signal lines. They record your preference for compliant operators.",
    "For search and indexing, use noindex (meta tag or X-Robots-Tag header) on pages that may be crawled but shouldn't show in results. A page blocked in robots.txt can still be indexed from links.",
    "Check crawler identity in your logs: match each claimed AI user agent to the operator's published IP ranges (OpenAI, Google and others publish them) or to reverse DNS followed by a forward lookup. Treat mismatches as unverified scrapers.",
    "Re-check monthly: fetch a protected URL from outside your network with no cookies and a spoofed crawler user agent. You should get a 401/403, not a 200."
  ],
  "sections": [
    {
      "heading": "A preference is not a lock",
      "points": [
        "RFC 9309 (the Robots Exclusion Protocol) says crawlers are requested to honor the rules and that the rules are not a form of access authorization. The file only affects clients that choose to read and obey it.",
        "Training opt-outs work the same way. Google-Extended, Applebot-Extended, Content-Signal: ai-train=no and TDM reservation headers tell compliant operators how they may use content. They don't stop a request from being answered.",
        "Google's robots.txt guide says the file is mainly for managing crawl traffic and that keeping a page out of results takes noindex or password protection. A URL blocked by robots.txt can still appear in results if other pages link to it.",
        "If a page returns 200 to an anonymous GET, assume it can be read, cached, archived and used for training by someone who ignores your preferences."
      ]
    },
    {
      "heading": "What to use instead for pages that must stay private",
      "points": [
        "Authentication at the origin is the lock: session cookies, SSO, mTLS for service APIs, or short-lived signed URLs for downloads. Rely on it rather than on obscure URLs or robots.txt.",
        "A CDN or WAF rule stops unwanted crawler classes before they reach your server, for example a managed AI-bot block, a custom rule that only lets verified bots through to /docs, or rate limits per IP or ASN. Edge rules are enforcement; robots.txt is a notice.",
        "Use noindex for pages that may be fetched but shouldn't be listed. For non-HTML files such as PDFs, send an X-Robots-Tag: noindex header.",
        "Keep sensitive paths out of sitemaps, llms.txt, agents.txt and robots.txt. Discovery files are public announcements."
      ]
    },
    {
      "heading": "Verify crawler identity in your logs",
      "points": [
        "User-agent strings are trivial to fake. Before you count a hit as GPTBot or Googlebot, check that the IP is in the operator's published range, or do a reverse DNS lookup on the IP, confirm the hostname is in the operator's domain, then do a forward lookup back to the same IP.",
        "OpenAI publishes separate IP lists for GPTBot, OAI-SearchBot and ChatGPT-User. Google documents both the DNS method and JSON IP range files for its crawlers and fetchers.",
        "Web Bot Auth (HTTP Message Signatures with a published key directory) lets an operator sign its requests. Cloudflare and Google document verifying them. A valid signature ties the request to the operator; a missing one means the request is unverified.",
        "Log the result as verified, unverified-claimed or unknown. Unverified traffic using a well-known crawler name is a scraper and a good candidate for a WAF rule."
      ]
    },
    {
      "heading": "What can go wrong",
      "points": [
        "You add Disallow: /admin/ and /exports/ to robots.txt and leave them unauthenticated; scrapers read the file and fetch both.",
        "You rely on Google-Extended to keep a pricing sheet private; it only covers Google's use of content it fetches, and other crawlers still get the page.",
        "You block GPTBot by user agent at the WAF, and a scraper using a generic browser user agent fetches the page anyway.",
        "You trust every request that says Googlebot and exempt it from rate limits; spoofers get unlimited access.",
        "You put a page behind robots.txt Disallow to hide it from search; it's indexed from inbound links with no snippet, and noindex can't be seen because the page is never crawled."
      ]
    }
  ],
  "checks": [],
  "ask_rightshold": {
    "start": "https://www.rightshold.com/start?ref=seo",
    "mcp_endpoint": "https://www.rightshold.com/mcp",
    "ask": "What does example.com's robots.txt and other AI-use signals say to AI crawlers, and which training opt-outs are in place, so I can see what's only a preference and still needs a real access control?",
    "mcp_call": {
      "tool": "read_ai_policy",
      "arguments": {
        "url": "https://example.com"
      }
    },
    "also": [
      "Free, no key, cached about 6 hours. It summarizes robots.txt per AI crawler and user-fetch agent, Content-Signal lines, ai.txt, TDMRep, and whether llms.txt exists. Those are the preference layer; it does not test your authentication or WAF.",
      "How to verify which AI agent is hitting your site with Web Bot Auth covers signature checks for agents that sign their requests.",
      "How to tell ChatGPT-User and other AI assistant fetches from AI training crawlers in server logs covers matching user agents to published IP lists."
    ],
    "scope": "Rightshold reads public robots.txt and AI-use signals on request. It does not enforce them, sit in front of your site, verify crawler IPs for you, or test your authentication or CDN rules. Preferences are not access control. Not legal advice."
  },
  "sources": [
    {
      "title": "RFC 9309: Robots Exclusion Protocol (rules are not access authorization)",
      "url": "https://www.rfc-editor.org/rfc/rfc9309.html"
    },
    {
      "title": "Google Search Central: Introduction to robots.txt (use noindex or password protection to keep pages out)",
      "url": "https://developers.google.com/search/docs/crawling-indexing/robots/intro"
    },
    {
      "title": "Google: Verify requests from Google crawlers and fetchers",
      "url": "https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot"
    },
    {
      "title": "OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, published IP lists)",
      "url": "https://developers.openai.com/api/docs/bots"
    },
    {
      "title": "Cloudflare docs: Web Bot Auth",
      "url": "https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/"
    },
    {
      "title": "Cloudflare docs: Verified bots",
      "url": "https://developers.cloudflare.com/bots/concepts/bot/verified-bots/"
    },
    {
      "title": "Content Signals Policy",
      "url": "https://contentsignals.org/"
    }
  ],
  "related": [
    {
      "title": "How to block AI training crawlers in robots.txt without leaving AI search",
      "url": "https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search"
    },
    {
      "title": "How to verify which AI agent is hitting your site with Web Bot Auth",
      "url": "https://www.rightshold.com/answers/how-to-verify-which-ai-agent-is-hitting-your-site-with-web-bot-auth"
    },
    {
      "title": "How to tell ChatGPT-User and other AI assistant fetches from AI training crawlers in server logs",
      "url": "https://www.rightshold.com/answers/how-to-tell-chatgpt-user-fetches-from-ai-training-crawlers-in-server-logs"
    },
    {
      "title": "How to configure robots.txt and agents.txt for AI agent discovery without blocking",
      "url": "https://www.rightshold.com/answers/how-to-configure-robots-txt-and-agents-txt-for-ai-agent-discovery-without-blocking"
    }
  ],
  "computed_from": "Curated guidance; license verdicts computed from Rightshold's license profiles at render time. Not legal advice.",
  "updated": "2026-11-07"
}