Rightshold™ Sign up

Answers · AI crawlers

How do I make my website readable to AI agents with llms.txt and markdown versions of pages?

Last updated

TL;DR: Serve the same content in a form agents can read cheaply, and make it easy to find. The llms.txt proposal (Jeremy Howard, first published September 2024, now in a v2) puts a short markdown file at /llms.txt with a one-line summary of the site and links to the pages that matter, and suggests a clean markdown copy of each page at the same URL with .md added. Link them with rel="alternate" type="text/markdown" for the markdown copy and rel="describedby" for the llms.txt, as HTML link tags or an HTTP Link header; optionally also answer Accept: text/markdown with markdown and send Vary: Accept. Be clear about what this buys you. It's a convention, not a standard every agent reads; Chrome's Lighthouse checks for llms.txt in its agentic browsing audits and coding agents often use it for docs, but Google says you don't need AI text files or special markup to appear in AI Overviews or AI Mode, where normal indexable HTML is what counts. So keep the HTML crawlable and complete, and treat llms.txt and markdown twins as a cheaper second door for agents, not a ranking trick.

What should I check, step by step?

1. Start with the HTML: important content in the server-rendered page as text (not only in client-side JavaScript or images), robots.txt allowing the crawlers you want, and nothing at the CDN or firewall quietly blocking them.

  1. Write /llms.txt in markdown: an H1 with the site or project name, a short blockquote summary, then H2 sections of links with a few words each. Point the links at the markdown versions, keep it short, and put rarely needed pages under an Optional section.
  2. Publish markdown twins: the same content as the HTML page, without navigation, ads or cookie banners, at page.md (or page.html.md, or index.md for directory URLs), served as text/markdown.
  3. Link them: <link rel="alternate" type="text/markdown" href="/page.md"> and <link rel="describedby" href="/llms.txt"> in the HTML head, or an HTTP Link header with the same relations, which also works on the markdown files themselves.
  4. Optional content negotiation: return markdown when the request's Accept header prefers text/markdown, and send Vary: Accept so caches keep the HTML and markdown apart. On Cloudflare Pro, Business or Enterprise, Markdown for Agents can do this conversion at the edge.
  5. Keep facts in sync: generate the HTML, markdown and any JSON from one source so the twins can't drift. A stale markdown copy is worse than none, since agents will quote it.
  6. For structured facts (prices, specs, API operations), also publish machine formats agents already parse: schema.org JSON-LD that matches the visible text, an OpenAPI description for APIs, and a sitemap that lists every public page.
  7. Test like an agent: curl /llms.txt and a few .md URLs, confirm 200 and text/markdown, follow every link in llms.txt, and check what each AI crawler is allowed in robots.txt.

What llms.txt is and isn't

It's a proposal at llmstxt.org, not an IETF or W3C standard. The v2 text says thousands of sites publish one, documentation platforms generate them automatically, and OpenAI, Anthropic and Google's Gemini API publish llms.txt files for their own developer docs.

Where it pays off

Agents that already know to look: coding agents and doc tools reading library and API documentation, and assistants whose users paste a docs URL. A tight llms.txt with markdown links saves them tokens and wrong guesses.

How do I ask Rightshold about this (free, no key)?

Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Does example.com have an llms.txt, and what do its robots.txt, ai.txt and TDM signals tell AI agents and crawlers?”

One-click setup

{
  "tool": "read_ai_policy",
  "arguments": {
    "domain": "example.com"
  }
}

Rightshold reads a site's public AI and crawl signals on request. It does not build, host or validate your llms.txt or markdown pages, and it doesn't affect how any search engine ranks you. Not legal advice.

What are the sources?

Related

Not legal advice. JSON · Markdown · All answers

Get a free API key in one call

No card and no email needed. 100 free credits; the license and policy checks stay free without a key.

curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_...   # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
  -H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'
Get a keyQuickstart