Rightshold™ Sign up

Answers · AI crawlers

How do I configure robots.txt and agents.txt for AI agent discovery without blocking legitimate agents?

Last updated

TL;DR: Treat discovery files as public front doors, not secrets. Keep /robots.txt, /agents.txt, /agents.json, /llms.txt, and well-known agent cards (for example /.well-known/agent-card.json) reachable to the crawlers and agents you want, while still disallowing training bots you do not want. The draft agents.txt standard (v1.0) says sites SHOULD Allow: /agents.txt in the default User-agent: * group of robots.txt, serve agents.txt as text/plain with CORS *, and put capability URLs (MCP:, A2A:, Skills:) in agents.txt without wallet addresses or API keys. Do not Disallow: / or a broad /* rule that also hides those paths, and do not invent a non-standard Agents-Txt: robots directive — the file lives at the canonical /agents.txt path. Separately, verify anything you consume from another site before connecting or paying: fetch over HTTPS, check Content-Type, confirm URLs stay on expected hosts, and validate A2A agent cards (and signatures when present) the same way you would any untrusted JSON. read_ai_policy summarizes what a site's robots.txt and related AI signals say; scan_text can check card or agents.txt prose for hidden instructions.

What should I check, step by step?

1. Publish /agents.txt at the site root (UTF-8, Content-Type: text/plain; charset=utf-8, Access-Control-Allow-Origin: *, Cache-Control public with a modest max-age). Serve /agents.json as application/json with the same CORS if you offer the structured companion.

  1. In robots.txt, under User-agent: *, explicitly Allow: /agents.txt, Allow: /agents.json, Allow: /llms.txt, and Allow: /.well-known/agent-card.json (and other discovery paths you publish) before any broad Disallow rules that could shadow them.
  2. Keep training opt-outs on the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) without Disallowing the discovery paths those operators need to read your policy — a 5xx or unreachable robots.txt is worse than a clear file.
  3. Put only capability announcements in agents.txt: Protocols:, Authorization:, MCP:, Skills:, A2A:, UCP:, WebMCP: as URLs or registered identifiers. No wallets, JWKs, API keys, or secrets.
  4. Point A2A: lines at HTTPS AgentCard URLs; keep full agent metadata in the card, not duplicated in agents.txt.
  5. Before your agent trusts another origin's agents.txt / agents.json / agent card: fetch HTTPS only, reject unexpected Content-Type, allowlist hostnames on every URL in the file, and validate card required fields (and JWS if present).
  6. Test with curl from outside your VPN: 200 on /robots.txt, /agents.txt, /agents.json, /llms.txt, and the card URL; confirm Allow lines match; confirm a training Disallow still works.
  7. Re-check weekly: read_ai_policy on your own domain, and re-fetch peer discovery files before renewing trust.

How do I keep agent discovery reachable in robots.txt?

RFC 9309 robots.txt rules are not access authorization, but major AI operators say their automatic crawlers follow them. A User-agent: * Disallow: / that you meant for scrapers also hides /agents.txt unless a more specific Allow wins for groups that match — put explicit Allow lines for discovery paths in the groups that matter.

What does agents.txt declare, and what does it not?

agents.txt is a draft community capability announcement (plain-text directives at /agents.txt, optional /agents.json catalog). It sits above robots.txt / sitemap / llms.txt as a signal of what agents can do: pay, authenticate, connect to MCP, load skills, or find A2A cards.

How do I verify discovery files before connecting or paying?

A file that claims MCP: https://evil.example/mcp is only text. Resolve and allowlist hosts; prefer pinning expected endpoints out-of-band for high-value counterparties.

What can go wrong?

Disallow: / on * with no Allow for /agents.txt; agents never see your MCP or AgentCard URLs.

How do I ask Rightshold about this (free, no key)?

Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “What does example.com's robots.txt tell AI crawlers, and does it leave /agents.txt, /llms.txt and related discovery paths reachable while training is opted out?”

One-click setup

{
  "tool": "read_ai_policy",
  "arguments": {
    "url": "https://example.com"
  }
}

Rightshold reads public robots.txt and AI-use signals on request and can scan text you paste. It does not publish your agents.txt, change your CDN rules, enforce robots.txt, or vouch that a remote agents.txt dialect is safe. Discovery files are announcements, not authorization. Not legal advice.

What are the sources?

Related

Not legal advice. JSON · Markdown · All answers

Get a free API key in one call

No card and no email needed. 100 free credits; the license and policy checks stay free without a key.

curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_...   # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
  -H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'
Get a keyQuickstart