Why isn't an AI-training opt-out or robots.txt preference access control, and what should I use instead?

What should I check, step by step?

1. Sort your URLs into three groups: public and fine for AI training, public but not for training (an opt-out is enough here), and must-not-be-read (needs a lock, not a preference).

2. Anything in the must-not-be-read group goes behind authentication: a login session, SSO, HTTP auth on staging, or short-lived signed URLs for files. Anonymous requests should get a 401, 403 or 404 instead of the content.
3. Don't list secret paths in robots.txt. Anyone can read the file, so a Disallow: /internal-reports/ line tells scrapers where to look.
4. Add CDN/WAF rules for crawler classes you don't want at all, for example Cloudflare's AI-bot blocking or a custom rule on path plus verified-bot status, so the request stops at the edge.
5. Keep training opt-outs in robots.txt (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and, if you use them, Content-Signal lines. They record your preference for compliant operators.
6. For search and indexing, use noindex (meta tag or X-Robots-Tag header) on pages that may be crawled but shouldn't show in results. A page blocked in robots.txt can still be indexed from links.
7. Check crawler identity in your logs: match each claimed AI user agent to the operator's published IP ranges (OpenAI, Google and others publish them) or to reverse DNS followed by a forward lookup. Treat mismatches as unverified scrapers.
8. Re-check monthly: fetch a protected URL from outside your network with no cookies and a spoofed crawler user agent. You should get a 401/403, not a 200.

Is an AI-training opt-out or robots.txt a lock?

RFC 9309 (the Robots Exclusion Protocol) says crawlers are requested to honor the rules and that the rules are not a form of access authorization. The file only affects clients that choose to read and obey it.

• Training opt-outs work the same way. Google-Extended, Applebot-Extended, Content-Signal: ai-train=no and TDM reservation headers tell compliant operators how they may use content. They don't stop a request from being answered.
• Google's robots.txt guide says the file is mainly for managing crawl traffic and that keeping a page out of results takes noindex or password protection. A URL blocked by robots.txt can still appear in results if other pages link to it.
• If a page returns 200 to an anonymous GET, assume it can be read, cached, archived and used for training by someone who ignores your preferences.

What should I use instead for pages that must stay private?

Authentication at the origin is the lock: session cookies, SSO, mTLS for service APIs, or short-lived signed URLs for downloads. Rely on it rather than on obscure URLs or robots.txt.

• A CDN or WAF rule stops unwanted crawler classes before they reach your server, for example a managed AI-bot block, a custom rule that only lets verified bots through to /docs, or rate limits per IP or ASN. Edge rules are enforcement; robots.txt is a notice.
• Use noindex for pages that may be fetched but shouldn't be listed. For non-HTML files such as PDFs, send an X-Robots-Tag: noindex header.
• Keep sensitive paths out of sitemaps, llms.txt, agents.txt and robots.txt. Discovery files are public announcements.

How do I verify crawler identity in my logs?

User-agent strings are trivial to fake. Before you count a hit as GPTBot or Googlebot, check that the IP is in the operator's published range, or do a reverse DNS lookup on the IP, confirm the hostname is in the operator's domain, then do a forward lookup back to the same IP.

• OpenAI publishes separate IP lists for GPTBot, OAI-SearchBot and ChatGPT-User. Google documents both the DNS method and JSON IP range files for its crawlers and fetchers.
• Web Bot Auth (HTTP Message Signatures with a published key directory) lets an operator sign its requests. Cloudflare and Google document verifying them. A valid signature ties the request to the operator; a missing one means the request is unverified.
• Log the result as verified, unverified-claimed or unknown. Unverified traffic using a well-known crawler name is a scraper and a good candidate for a WAF rule.

What can go wrong?

You add Disallow: /admin/ and /exports/ to robots.txt and leave them unauthenticated; scrapers read the file and fetch both.

• You rely on Google-Extended to keep a pricing sheet private; it only covers Google's use of content it fetches, and other crawlers still get the page.
• You block GPTBot by user agent at the WAF, and a scraper using a generic browser user agent fetches the page anyway.
• You trust every request that says Googlebot and exempt it from rate limits; spoofers get unlimited access.
• You put a page behind robots.txt Disallow to hide it from search; it's indexed from inbound links with no snippet, and noindex can't be seen because the page is never crawled.