# How do I make my website readable to AI agents with llms.txt and markdown versions of pages?

**Short answer:** Serve the same content in a form agents can read cheaply, and make it easy to find. The llms.txt proposal (Jeremy Howard, first published September 2024, now in a v2) puts a short markdown file at /llms.txt with a one-line summary of the site and links to the pages that matter, and suggests a clean markdown copy of each page at the same URL with .md added. Link them with rel="alternate" type="text/markdown" for the markdown copy and rel="describedby" for the llms.txt, as HTML link tags or an HTTP Link header; optionally also answer Accept: text/markdown with markdown and send Vary: Accept. Be clear about what this buys you. It's a convention, not a standard every agent reads; Chrome's Lighthouse checks for llms.txt in its agentic browsing audits and coding agents often use it for docs, but Google says you don't need AI text files or special markup to appear in AI Overviews or AI Mode, where normal indexable HTML is what counts. So keep the HTML crawlable and complete, and treat llms.txt and markdown twins as a cheaper second door for agents, not a ranking trick.

## Checklist
1. Start with the HTML: important content in the server-rendered page as text (not only in client-side JavaScript or images), robots.txt allowing the crawlers you want, and nothing at the CDN or firewall quietly blocking them.
2. Write /llms.txt in markdown: an H1 with the site or project name, a short blockquote summary, then H2 sections of links with a few words each. Point the links at the markdown versions, keep it short, and put rarely needed pages under an Optional section.
3. Publish markdown twins: the same content as the HTML page, without navigation, ads or cookie banners, at page.md (or page.html.md, or index.md for directory URLs), served as text/markdown.
4. Link them: <link rel="alternate" type="text/markdown" href="/page.md"> and <link rel="describedby" href="/llms.txt"> in the HTML head, or an HTTP Link header with the same relations, which also works on the markdown files themselves.
5. Optional content negotiation: return markdown when the request's Accept header prefers text/markdown, and send Vary: Accept so caches keep the HTML and markdown apart. On Cloudflare Pro, Business or Enterprise, Markdown for Agents can do this conversion at the edge.
6. Keep facts in sync: generate the HTML, markdown and any JSON from one source so the twins can't drift. A stale markdown copy is worse than none, since agents will quote it.
7. For structured facts (prices, specs, API operations), also publish machine formats agents already parse: schema.org JSON-LD that matches the visible text, an OpenAPI description for APIs, and a sitemap that lists every public page.
8. Test like an agent: curl /llms.txt and a few .md URLs, confirm 200 and text/markdown, follow every link in llms.txt, and check what each AI crawler is allowed in robots.txt.

## What llms.txt is and isn't
- It's a proposal at llmstxt.org, not an IETF or W3C standard. The v2 text says thousands of sites publish one, documentation platforms generate them automatically, and OpenAI, Anthropic and Google's Gemini API publish llms.txt files for their own developer docs.
- It isn't a permission or opt-out file. Whether crawlers may train on or index your content is still set in robots.txt (and, where you use them, Content-Signal lines or TDM reservations); llms.txt only tells an agent what to read.
- Lighthouse's llms.txt audit is marked not applicable when the file is missing, because Chrome treats it as optional for now.

## Where it pays off
- Agents that already know to look: coding agents and doc tools reading library and API documentation, and assistants whose users paste a docs URL. A tight llms.txt with markdown links saves them tokens and wrong guesses.
- User-triggered fetches: when ChatGPT-User or Claude-User fetches one page for a question, a clean page (or markdown twin) with the answer near the top is what gets read. Long navigation and script-built content get in the way.
- It won't fix Google rankings on its own. Google's guidance for AI features is the usual SEO basics: indexable pages with snippets, internal links, text content and structured data that matches the page.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "Does example.com have an llms.txt, and what do its robots.txt, ai.txt and TDM signals tell AI agents and crawlers?"

```json
{
  "tool": "read_ai_policy",
  "arguments": {
    "domain": "example.com"
  }
}
```
- Free, no key, cached 6 hours. It reports whether /llms.txt exists, along with robots.txt rules per AI crawler and user-fetch agent, Content-Signal lines, ai.txt and W3C TDMRep.
- It checks that llms.txt is there, not whether its links work or its markdown matches your pages; run the curl checks above for that.
- Rightshold's own answer pages follow this pattern: each has .md and .json twins, Link headers with rel=alternate and rel=describedby, Vary: Accept, and a listing in /llms.txt.

_Rightshold reads a site's public AI and crawl signals on request. It does not build, host or validate your llms.txt or markdown pages, and it doesn't affect how any search engine ranks you. Not legal advice._

## Sources
- [llmstxt.org: The /llms.txt file (proposal, v2)](https://llmstxt.org/)
- [Chrome for Developers: Lighthouse llms.txt audit (agentic browsing)](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt)
- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [Cloudflare docs: Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)
- [MDN: Content negotiation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Content_negotiation)

## Related
- [How to tell ChatGPT-User and other AI assistant fetches from AI training crawlers in server logs](https://www.rightshold.com/answers/how-to-tell-chatgpt-user-fetches-from-ai-training-crawlers-in-server-logs)
- [How to write MCP tool descriptions so agents pick the right tool](https://www.rightshold.com/answers/how-to-write-mcp-tool-descriptions-so-agents-pick-the-right-tool)
- [Does a no-AI notice on your art do anything legally?](https://www.rightshold.com/answers/does-a-no-ai-notice-on-your-art-do-anything-legally)

_Not legal advice. Updated 2026-10-30. JSON: https://www.rightshold.com/answers/how-to-make-a-website-readable-to-ai-agents-with-llms-txt-and-markdown.json_