# How do I configure robots.txt and agents.txt for AI agent discovery without blocking legitimate agents?

**Short answer:** Treat discovery files as public front doors, not secrets. Keep /robots.txt, /agents.txt, /agents.json, /llms.txt, and well-known agent cards (for example /.well-known/agent-card.json) reachable to the crawlers and agents you want, while still disallowing training bots you do not want. The draft agents.txt standard (v1.0) says sites SHOULD Allow: /agents.txt in the default User-agent: * group of robots.txt, serve agents.txt as text/plain with CORS *, and put capability URLs (MCP:, A2A:, Skills:) in agents.txt without wallet addresses or API keys. Do not Disallow: / or a broad /* rule that also hides those paths, and do not invent a non-standard Agents-Txt: robots directive — the file lives at the canonical /agents.txt path. Separately, verify anything you consume from another site before connecting or paying: fetch over HTTPS, check Content-Type, confirm URLs stay on expected hosts, and validate A2A agent cards (and signatures when present) the same way you would any untrusted JSON. read_ai_policy summarizes what a site's robots.txt and related AI signals say; scan_text can check card or agents.txt prose for hidden instructions.

## Checklist
1. Publish /agents.txt at the site root (UTF-8, Content-Type: text/plain; charset=utf-8, Access-Control-Allow-Origin: *, Cache-Control public with a modest max-age). Serve /agents.json as application/json with the same CORS if you offer the structured companion.
2. In robots.txt, under User-agent: *, explicitly Allow: /agents.txt, Allow: /agents.json, Allow: /llms.txt, and Allow: /.well-known/agent-card.json (and other discovery paths you publish) before any broad Disallow rules that could shadow them.
3. Keep training opt-outs on the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) without Disallowing the discovery paths those operators need to read your policy — a 5xx or unreachable robots.txt is worse than a clear file.
4. Put only capability announcements in agents.txt: Protocols:, Authorization:, MCP:, Skills:, A2A:, UCP:, WebMCP: as URLs or registered identifiers. No wallets, JWKs, API keys, or secrets.
5. Point A2A: lines at HTTPS AgentCard URLs; keep full agent metadata in the card, not duplicated in agents.txt.
6. Before your agent trusts another origin's agents.txt / agents.json / agent card: fetch HTTPS only, reject unexpected Content-Type, allowlist hostnames on every URL in the file, and validate card required fields (and JWS if present).
7. Test with curl from outside your VPN: 200 on /robots.txt, /agents.txt, /agents.json, /llms.txt, and the card URL; confirm Allow lines match; confirm a training Disallow still works.
8. Re-check weekly: read_ai_policy on your own domain, and re-fetch peer discovery files before renewing trust.

## How do I keep agent discovery reachable in robots.txt?
- RFC 9309 robots.txt rules are not access authorization, but major AI operators say their automatic crawlers follow them. A User-agent: * Disallow: / that you meant for scrapers also hides /agents.txt unless a more specific Allow wins for groups that match — put explicit Allow lines for discovery paths in the groups that matter.
- The agents.txt draft (§4.3) says a compliant site SHOULD include Allow: /agents.txt in the default * group, and MUST NOT emit a separate Agents-Txt: robots directive: the path is fixed at /agents.txt, unlike Sitemap: which may point anywhere.
- How to block AI training crawlers in robots.txt without leaving AI search covers training vs search tokens. This page is the discovery half: do not let a blunt crawl policy make agents unable to find your MCP endpoint, AgentCard, or payment protocol announcement.
- Keep robots.txt itself reachable (2xx). Under RFC 9309, many 4xx responses are treated as "no robots.txt" (everything allowed), and 5xx can be treated as "assume disallow" — either extreme breaks a careful Allow/Disallow split.

## What does agents.txt declare, and what does it not?
- agents.txt is a draft community capability announcement (plain-text directives at /agents.txt, optional /agents.json catalog). It sits above robots.txt / sitemap / llms.txt as a signal of what agents can do: pay, authenticate, connect to MCP, load skills, or find A2A cards.
- It is not a single IETF standard yet, and other projects have used the same filename with incompatible formats. When you publish, link your dialect (for example the agents-txt.com standard URL in a comment). When you consume, detect which shape you got before parsing directives as gospel.
- Security rules from the draft: no wallet or treasury addresses, no API keys, no JWKs, no credentials. Payment details stay in 402 responses; auth details stay at protocol discovery endpoints such as /.well-known/agent-configuration.
- Serving requirements in the draft: CORS * so browser-context agents can fetch the file; prefer Cache-Control: public, max-age=3600 unless you change capabilities more often.

## How do I verify discovery files before connecting or paying?
- A file that claims MCP: https://evil.example/mcp is only text. Resolve and allowlist hosts; prefer pinning expected endpoints out-of-band for high-value counterparties.
- For A2A AgentCards, follow How to validate an A2A agent card: HTTPS fetch from the expected path, required fields for the claimed A2A version, HTTPS endpoint URLs, and verify JWS signatures when present (A2A says clients should verify at least one signature before trusting a card).
- Scan free-text fields (descriptions, skill blurbs, agents.txt comments) with the same prompt-injection caution you use for any untrusted tool output — scan_text can flag hidden instructions aimed at your agent.
- read_ai_policy on a domain reports robots.txt per AI user agent, Content-Signal lines, ai.txt / TDM signals, and whether llms.txt exists. Use it to confirm your own Allow/Disallow intent and to inspect a peer before you automate against them.

## What can go wrong?
- Disallow: / on * with no Allow for /agents.txt; agents never see your MCP or AgentCard URLs.
- agents.txt includes a wallet address or API key "for convenience"; scrapers harvest it.
- You parse a different project's agents.txt dialect as if it were the agents-txt.com v1.0 directive set and connect to the wrong URL.
- CDN or WAF blocks anonymous GETs to /.well-known/* while robots.txt claims Allow — assistants and agents get 403s.
- You trust an AgentCard URL from agents.txt without checking the card's signatures or that skills[].examples are free of injection.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "What does example.com's robots.txt tell AI crawlers, and does it leave /agents.txt, /llms.txt and related discovery paths reachable while training is opted out?"

```json
{
  "tool": "read_ai_policy",
  "arguments": {
    "url": "https://example.com"
  }
}
```
- Free, no key, cached about 6 hours. It summarizes robots.txt per AI crawler and user-fetch agent, Content-Signal lines, ai.txt, TDMRep, and whether llms.txt exists.
- scan_text can check an agents.txt body or AgentCard JSON for hidden instructions before your agent follows MCP: or A2A: URLs.
- precall_check before connecting to an MCP URL you discovered only from a third-party agents.txt.
- How to validate an A2A agent card covers required fields and signature checks for cards linked from A2A: lines.

_Rightshold reads public robots.txt and AI-use signals on request and can scan text you paste. It does not publish your agents.txt, change your CDN rules, enforce robots.txt, or vouch that a remote agents.txt dialect is safe. Discovery files are announcements, not authorization. Not legal advice._

## Sources
- [agents.txt Standard v1.0 (draft) — discovery, robots.txt Allow, security](https://agents-txt.com/spec)
- [agents-txt/agents-txt on GitHub](https://github.com/agents-txt/agents-txt)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
- [A2A Protocol Specification: Agent Card](https://a2a-protocol.org/latest/specification/)
- [How to block AI training crawlers in robots.txt without leaving AI search (Rightshold)](https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search)

## Related
- [How to block AI training crawlers in robots.txt without leaving AI search](https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search)
- [How to validate an A2A agent card](https://www.rightshold.com/answers/how-to-validate-an-a2a-agent-card)
- [How to make a website readable to AI agents with llms.txt and markdown pages](https://www.rightshold.com/answers/how-to-make-a-website-readable-to-ai-agents-with-llms-txt-and-markdown)
- [How to tell ChatGPT-User and other AI assistant fetches from AI training crawlers in server logs](https://www.rightshold.com/answers/how-to-tell-chatgpt-user-fetches-from-ai-training-crawlers-in-server-logs)

_Not legal advice. Updated 2026-11-06. JSON: https://www.rightshold.com/answers/how-to-configure-robots-txt-and-agents-txt-for-ai-agent-discovery-without-blocking.json_