What is the new "User-Agent: __usage_ai-training" line for robots.txt, which AI companies say they'll honor it and from when, and does it replace blocking GPTBot, ClaudeBot or Google-Extended?

What should I check, step by step?

1. Decide what you're opting out of. The line covers training only: using your content to change the parameters of a generative AI model. It isn't a block on AI search, on agents fetching a page for a user, or on Google Search or Bing results.

2. To opt out site-wide, add a separate group: "User-Agent: __usage_ai-training" then "Disallow: /". To keep a section trainable, add "Allow: /press-kit/" (or your path) to the same group; longest-match rules apply as for any robots.txt group.
3. Keep your existing crawler groups. A shared opt-out doesn't stop GPTBot, ClaudeBot or CCBot from fetching pages, and the draft says a Google-Extended or other company-specific rule isn't read as the shared preference. If you want a crawler kept out, name it.
4. Don't count on "User-agent: *". The draft says wildcard groups never match the policy token, so a site that blocks everything with "User-agent: *" has expressed no shared training preference.
5. If you want your pages cited in Gemini Apps answers, add an explicit Google-Extended group as well. Google says it will read a shared opt-out as a Google-Extended disallow only when Google-Extended isn't specified, and that sites that specifically allow Google-Extended are eligible to appear via grounding in Gemini Apps.
6. If you use Bing's NoArchive or NoCache tags to keep content out of AI, plan the move: Microsoft stops reading them for training or grounding after March 1, 2027, and grounding needs its own page-level setting (there's no robots.txt grounding preference). Sites using those tags on December 15, 2026 can record a temporary grounding preference in Bing Webmaster Tools.
7. Keep blocking or naming the training crawlers of companies that haven't committed. The October 30 announcements came from Google, Microsoft, OpenAI and Anthropic; the line does nothing for crawlers whose operators don't read it.
8. Keep a dated copy of robots.txt each time you change it. The draft ties a preference to the robots.txt in effect when a page was retrieved, so the date you added the line matters.

What does "User-Agent: __usage_ai-training" mean in robots.txt?

The draft, draft-kelley-aipref-training-preference-signal-00, describes a 'policy token': a name used on a robots.txt User-Agent line that isn't a crawler but a purpose. It defines one, __usage_ai-training, and introduces no new directives or grammar. Its example is a group with "User-Agent: __usage_ai-training", "Disallow: /" and "Allow: /press-kit/", which expresses a preference against AI training everywhere except under /press-kit/.

• A consumer looks up the path under that token using the normal RFC 9309 path matching. If the path is disallowed, that's a preference against training; if allowed, a preference in favor. If no group names __usage_ai-training, no preference has been expressed. Wildcard groups must not be read as matching the token, and a company-specific token such as Google-Extended must not be treated as expressing the shared preference.
• The meaning comes from the IETF AI Preferences working group's vocabulary draft (draft-ietf-aipref-vocab-08): AI Training is using an asset to modify the learned parameters of a generative AI model. Training models used only for non-generative tasks such as classification, ranking or scoring isn't included.

Does the __usage_ai-training line block AI crawlers from fetching my site?

It isn't a crawl rule. The draft says the token isn't a directive for crawling behavior, that crawlers keep reading the groups for their own product tokens exactly as before, and that a consumer that crawls only for training MAY skip pages with a preference against training, which is optional. If you want a crawler to stay away, you still need a group that names it.

• It isn't enforcement. Like the rest of robots.txt it's a published preference, and the draft says private agreements may override it. It applies to content retrieved while the line is in your file, so it doesn't reach copies already collected or models already trained.
• It isn't a standard yet. The IETF Datatracker lists it as an individual Internet-Draft that isn't endorsed by the IETF and has no formal standing in the standards process. The author told the working group's mailing list he was proposing it as input to the group's separate draft on attaching preferences to content (draft-ietf-aipref-attach-05, which would add a Content-Usage robots.txt rule and HTTP header), so the syntax could still change.

Which AI companies will honor the __usage_ai-training robots.txt line, and when?

Google (Search Central blog, October 30, 2026): it plans to support the control in 2027, and its common crawlers page says early 2027. When a site opts out with the token and doesn't specify Google-Extended, Google plans to interpret that as if Google-Extended were disallowed. Google's crawler documentation says Google-Extended governs both training of future Gemini models and grounding in Gemini Apps and Grounding with Google Search on Vertex AI, and doesn't affect inclusion or ranking in Google Search. It doesn't touch AI Overviews or AI Mode, which run on Googlebot.

• Microsoft (Bing blog, October 30, 2026): it will honor the shared robots.txt preference from February 1, 2027 as it crawls, adds a page-level training setting (a Bingbot meta tag or X-Robots-Tag with ai-training:disallow) that overrides robots.txt, and adds a separate page-level AI Grounding preference with Allow, DisallowExtended (Microsoft's own experiences only, not grounding it sells to third parties) and Disallow. Both default to Allow, neither changes eligibility for Bing search results, and NoArchive and NoCache stop counting as AI preferences after March 1, 2027.
• OpenAI (October 30, 2026): it will honor the signal, aims to implement it in the coming months, and says it won't replace its existing robots.txt controls. Anthropic: the draft's author, an Anthropic engineer, wrote on the IETF list that Anthropic has already started implementing support, complementing its existing crawler controls, and that how it interprets the signals may change as standards evolve.