Do AI companies have to obey robots.txt and other AI opt-outs when they crawl sites for training in the EU, under the AI Act and its General-Purpose AI Code of Practice, and what can a site owner do if they don't?

What should I check, step by step?

1. Put your reservation in robots.txt with groups that name each training crawler you want out (for example GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent). robots.txt per RFC 9309 is the one protocol the Code names.

2. Consider adding the shared "User-Agent: __usage_ai-training" group too, which Google, Microsoft, OpenAI and Anthropic said in October 2026 they would honor; Google's announcement says the approach is consistent with the Code. It isn't read by anyone yet.
3. Don't rely on terms of use alone. The Hamburg Higher Regional Court held in December 2025 that a reservation written only in natural language on a website wasn't a machine-readable one for a 2021 crawl. Keep the terms, but also publish a machine-readable signal.
4. If you publish files or images that get copied elsewhere, consider an asset-based signal as well (for example TDMRep or embedded metadata). The Code mentions asset-based and location-based metadata, but commits signatories only to protocols that are standardized or widely adopted, so robots.txt remains the safest base.
5. Keep paywalled or members-only content behind a real login. Measure 1.2 commits signatories not to circumvent effective technological measures such as paywalls; a hidden page isn't one.
6. Find each provider's published crawler information and subscribe to updates. Measure 1.3(4) commits signatories to publish their crawlers, robots.txt features and other opt-out measures, with a way for rightsholders to be notified automatically when it changes.
7. Keep dated copies of your robots.txt and other signals, and evidence of any crawling after you opted out (server logs with verified crawler IPs).
8. If you think a signatory ignored your opt-out, use its designated point of contact and complaint mechanism (Measure 1.5) and keep the reply. Copyright claims go to national courts; talk to an EU copyright lawyer before relying on any of this.

Does the EU AI Act require AI companies to respect robots.txt opt-outs?

Article 53(1)(c) applies to providers of general-purpose AI models, the companies that train and offer models such as large language models, not to every website scraper. It requires a policy to comply with EU copyright law and in particular to identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed under Article 4(3) of Directive (EU) 2019/790. That article lets rightsholders opt out of the commercial text-and-data-mining exception, and says that for content made publicly available online the reservation should be made in an appropriate manner, such as machine-readable means.

• Recital 106 says any provider placing a general-purpose AI model on the EU market should comply regardless of the jurisdiction in which the copyright-relevant acts underpinning training take place, so a US company training in the US is covered if it offers the model in the EU. Article 53(1)(d) separately requires a public summary of the content used for training.
• Timing: these obligations have applied since August 2, 2025. The Commission's power to fine providers of general-purpose AI models (Article 101: up to 3% of total worldwide annual turnover or EUR 15 million, whichever is higher) started on August 2, 2026. Providers of models placed on the market before August 2, 2025 have until August 2, 2027 to comply.

What does the EU General-Purpose AI Code of Practice say about robots.txt?

Measure 1.3 of the Copyright chapter: signatories that crawl the web for text and data mining and training commit to employ crawlers that read and follow robots.txt as specified in RFC 9309 (and later versions the IETF shows are feasible), and to identify and comply with other appropriate machine-readable protocols for rights reservations, for example asset-based or location-based metadata, that are adopted by international or European standards bodies or are state of the art, widely adopted by rightsholders and generally agreed through an inclusive process. It adds that this doesn't limit rightsholders' right to reserve their rights in any appropriate manner.

• The same measure commits signatories to publish information about the crawlers they use, their robots.txt features and other measures they use to honor reservations, with a way for rightsholders to be notified automatically of updates, and encourages signatories that run search engines to make sure honoring an AI training opt-out doesn't directly hurt how the content is indexed in search. Measure 1.2 covers not circumventing paywalls and excluding sites courts or authorities have recognized as persistent commercial-scale infringers; Measure 1.5 requires a point of contact and a complaints mechanism.
• The Code is voluntary. Article 53(4) lets providers rely on it to demonstrate compliance; those who don't sign must show adequate alternative means. The Commission's list of signatories includes Amazon, Anthropic, Google, IBM, Microsoft, Mistral AI and OpenAI, among others; xAI signed up to the Safety and Security chapter only, so it must demonstrate copyright compliance another way.

Is a terms-of-use AI opt-out enough in the EU, or does it have to be machine-readable?

The commitments are framed around crawling to compile data for text and data mining and training. They don't say anything about an AI assistant fetching your page in real time for a user, and they bind providers of general-purpose models, not every company that runs a scraper. They also don't reach models trained on content collected before you published your opt-out.

• Courts are still working out what counts as machine-readable. In Kneschke v. LAION, the Hamburg Higher Regional Court (December 10, 2025, 5 U 104/24) dismissed a photographer's appeal: it held the dataset builder could rely on the scientific research exception, which can't be opted out of, and that a reservation written only in natural language in a site's terms of use wasn't machine-readable for the 2021 crawl in question. Germany's Federal Court of Justice heard the case on September 3, 2026, indicated it would refer questions to the EU Court of Justice, and set December 17, 2026 to announce its decision (I ZR 281/25).
• The practical answer is to make your opt-out as plainly machine-readable as you can, in the formats providers have committed to read, and keep evidence of when you published it. A site owner can't enforce the AI Act directly; the AI Office supervises providers, and copyright infringement is a matter for national courts.