Answers · AI crawlers
What is RSL (Really Simple Licensing), how do I add an RSL license to my site's robots.txt, and will AI crawlers actually read it or pay? Does it block anything on its own?
Last updated
TL;DR: RSL is an open XML format for telling AI crawlers and agents what they may do with your content and on what terms: which uses are permitted or prohibited (all, ai-all, ai-train, ai-input, ai-index, search), and what they owe (free, attribution, subscription, per crawl, per use, purchase and more). RSL 1.0 became a Recommendation on December 10, 2025. You publish a license file such as /license.xml and point to it with a robots.txt line, "License: https://example.com/license.xml"; an HTTP Link header (rel="license"), HTML, RSS and file metadata are equivalent ways to attach it. The License line doesn't change Allow or Disallow and doesn't block anything: it's a declaration that compliant clients must read before using your content. Enforcement comes from your CDN or bot management, and RSL's own Crawler Authorization Protocol expects a license token on each request. Adoption on the reading side is thin: a public scan found 4 of the top 1,000 sites with a License line on October 28, 2026, and no major AI model company has publicly committed to honor RSL files.
What should I check, step by step?
1. Write down what you actually want first: no AI use at all, no training but answers are fine, AI use only under a paid license, or free use with attribution. Each maps to a different RSL template.
- Create the license file. The simplest refusal is <prohibits type="usage">ai-train ai-input</prohibits> inside <content url="/"><license>…</license></content> under <rsl xmlns="https://rslstandard.org/rsl">. Use ai-all to cover every AI use, including indexing and uses not yet named.
- If you want to be paid, use <permits type="usage"> for the use and a <payment> element with its type (subscription, crawl, use, purchase, training, contribution) and either a link to your own licensing contact (<custom>) or a standard license or license server (<standard>, server="…").
- Don't list the same use as both permitted and prohibited. The spec says prohibitions take precedence when both appear for the same type, and that all covers any automated processing, AI training and search included. A file that permits ai-train and also prohibits all can be read as allowing nothing.
- Add "License: https://yourdomain/license.xml" to robots.txt as an absolute URL. Put it at the top of the file, before any User-agent line, so it's clearly global; a License line inside a group applies only to the crawlers that select that group, and for them it replaces the global one.
- Repeat it on every subdomain. RSL's getting-started guide says each subdomain needs its own robots.txt with its own License line.
- Make sure crawlers can fetch the file. Serve it as application/rsl+xml where you can, and exempt it from bot challenges: a license file behind a JavaScript challenge returns an HTML block page to a crawler instead of your terms.
- Set max-age on the <rsl> root if your terms change. Clients may treat a license as current for that many days; the default is 30.
- Keep blocking or charging for crawlers in robots.txt, your CDN or a pay-per-crawl gate. RSL records your terms; it doesn't stop a crawler that ignores them.
- Keep a dated copy of each license file version and the robots.txt that pointed to it. If you later claim a crawler used content against your terms, that's the record of what the terms said and when.
What is the RSL License directive in robots.txt, and how do crawlers find an RSL license?
RSL 1.0 is published by the RSL Technical Steering Committee as an industry specification with Recommendation status, dated December 10, 2025. It defines an XML vocabulary in the https://rslstandard.org/rsl namespace, ways to associate a license with content, and protocols for acquiring licenses (the Open License Protocol, an OAuth 2.0 extension), proving them on requests (the Crawler Authorization Protocol) and encrypting licensed files.
- A license document has an <rsl> root with one or more <content> elements. Each <content> has a url (a path that follows robots.txt matching rules, wildcards included) and one or more <license> elements listing permits, prohibits, payment and optional legal terms. Where two <content> scopes overlap, the more specific one wins for those assets.
- Discovery works five ways that the spec treats as equivalent: the robots.txt License directive, an HTTP Link header with rel="license" and type application/rsl+xml, HTML <link> or <script> elements, an RSS module and embedded file metadata. Clients are told to check all of them. If none is found, the resource counts as unlicensed under RSL.
What do ai-train, ai-input, ai-index and search mean in an RSL license?
The usage vocabulary is short. all is any automated processing, including AI training and search. ai-all is any use by AI systems, including training, inference, indexing, retrieval, generation, grounding or evaluation. ai-train is training or fine-tuning; ai-input is input to AI models, such as retrieval-augmented generation, grounding or AI search summaries; ai-index is inclusion in an AI system's index or retrieval database; search is building a search index and returning links and short excerpts, and explicitly excludes AI-generated summaries.
- If a <permits> element exists for a type, only the listed values are allowed for that type. <prohibits> overrides, so a use named in both is prohibited. That's how RSL can say "search yes, AI summaries no", something a single robots.txt Disallow can't.
- Payment types include free, attribution, subscription, purchase, training, crawl, use and contribution, optionally with an <amount> and currency. A <standard> element can point to a known license; RSL's templates use a Creative Commons or OSI license URL that way. If a <content> names a license server, clients must get a license token from it before access, even when the license is free.
Do AI companies honor RSL licenses, and does RSL block crawlers?
On the publisher side, RSL says more than 1,500 organizations endorse it, and the CDNs Fastly, Cloudflare and Akamai said at the 1.0 release that they support it. Publishing is still rare: a third-party scan of the top 1,000 sites found 4 with a License line on October 28, 2026, and only 2 serving a license file a crawler could read. On November 11, 2026 we found License lines in the robots.txt files of The Guardian, Bild and Medium; Bild's file prohibits ai-train and ai-input, and Medium's returned a bot-challenge page with HTTP 403 to a plain request.
- On the reading side, we found no public commitment from a major AI model company to honor RSL files as of November 2026, and a public scan reached the same conclusion. The spec's MUSTs bind clients that choose to implement RSL; a crawler that doesn't read the file isn't stopped by it.
- That's why the spec pairs it with enforcement. Under the Crawler Authorization Protocol a licensed crawler sends "Authorization: License <token>", and a server without a valid token answers 401 or 402 with a Link to the governing license. The spec says this should be paired with bot management or Web Bot Auth that identifies the crawler, because user-agent names can be copied.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “A site's RSL license.xml permits all uses under a standard CC BY 4.0 license. Can our crawler use those pages to train a commercial model, and what do we owe the site?”
{
"tool": "check_license",
"arguments": {
"license": "CC-BY-4.0",
"intended_use": {
"commercial_use": true,
"model_training": true,
"attribution_planned": true
}
}
}- Free, no key. For CC BY 4.0 it answers needs_review for model_training (whether training creates an adaptation and how attribution applies to a model isn't settled), lists the duties (attribution, license notice, marking changes, no added restrictions) and asks for a human decision. It reads the named license, not the RSL file itself, so check the file's permits, prohibits and payment terms first.
- read_ai_policy {domain} reads the same site's robots.txt for 16 AI crawler tokens, Content-Signal lines, ai.txt and TDMRep. It doesn't parse RSL License lines or license files yet; open the License URL yourself.
Rightshold reads public license and crawl signals. It isn't affiliated with the RSL Collective or any company named here, can't make a crawler read or honor an RSL file and doesn't operate an RSL license server. Terms are as written in the RSL 1.0 specification and RSL's guides, and the robots.txt and license files described were fetched on November 11, 2026; sites may change them. Not legal advice.
What are the sources?
- RSL: Really Simple Licensing (RSL) 1.0 Specification (Recommendation, December 10, 2025)
- RSL: Getting Started (license templates and robots.txt line)
- RSL: Adding RSL to robots.txt
- RSL: RSL and AI Preferences
- The Verge: RSL 1.0 has arrived, allowing publishers to ask AI companies to pay to scrape content (December 2025)
- Doccraft: RSL licensing for AI crawlers, 4 of the top 1,000 sites publish it (scan of October 28, 2026)
- IETF: RFC 9309, Robots Exclusion Protocol
Related
- How to charge AI crawlers per request with HTTP 402 without blocking search crawlers
- What is the __usage_ai-training robots.txt line, and which AI companies will honor it?
- Why an AI training opt-out or robots.txt is not access control
- How to verify which AI agent is hitting your site with Web Bot Auth
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'