Rightshold™ Sign up

Answers · AI crawlers

How do I catch AI crawlers and bots that ignore robots.txt, using a disallowed trap URL?

Last updated

TL;DR: Add a path to robots.txt as Disallow for every user agent, never link to it or put it in your sitemap, give it a name no one would guess, and log every request to it. A client that fetches that path almost certainly read your robots.txt and went where it was told not to; nothing else points there. A second trap catches crawlers that never read robots.txt: a disallowed URL that is linked from pages, so only link-followers that skip the robots.txt check reach it. Before you call a hit a violation, rule out three innocent cases. RFC 9309 lets crawlers cache robots.txt for up to 24 hours (OpenAI says its search systems can take about 24 hours to adjust to a change), so wait a day after adding the trap. User-triggered fetchers aren't bound the same way: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. And RFC 9309 notes that listing a path in robots.txt makes it discoverable, so security scanners and curious people will hit it too. Then identify the visitor: check the claimed bot's published IP list or reverse DNS, or verify a Web Bot Auth signature if the request carried one, which gate_check does for free.

What should I check, step by step?

1. Pick a trap path that looks plausible but is unique and unguessable (a random segment), add Disallow: /that-path/ under User-agent: *, and keep it out of the sitemap and all links.

  1. Make sure robots.txt returns 200. Under RFC 9309 a 4xx robots.txt lets crawlers access anything and a 5xx means they must assume complete disallow, so a broken file changes what counts as a violation.
  2. Serve the trap a plain 404 or an empty noindex page. Don't serve content worth taking, and don't redirect.
  3. Log the full request on a hit: time, IP, user agent, every header (including any Signature, Signature-Input and Signature-Agent) and whether the same IP fetched robots.txt shortly before.
  4. Wait at least 24 hours after publishing the trap before treating hits as violations; RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours, and Google says it generally caches it for up to 24 hours.
  5. Classify each hit: verified bot (IP list, reverse DNS or a valid signature), user-triggered fetcher (ChatGPT-User, Perplexity-User and similar, which say robots.txt may not apply), scanner or human, or unverified automation claiming a bot name.
  6. Optionally add a second, linked trap: a normal-looking link to another disallowed path. Keep it a real link a person could see; Google's spam policies treat links hidden with CSS, zero-size text or off-screen positioning as hidden link abuse when used to manipulate search.
  7. Act on verified identities only. Block or rate-limit a verified operator that keeps hitting the trap; for unverified hits, block the pattern at your edge and don't blame the bot whose name was borrowed.

Why is a robots.txt-disallowed trap URL better evidence than a user-agent string?

robots.txt is a request, not a lock: RFC 9309 says its rules are not a form of access authorization. Your server logs show who asked for robots.txt and which paths each client then fetched, but anyone can send any user-agent string, so a hit only tells you what a client claimed to be.

What innocent reasons can explain a hit on a robots.txt-disallowed trap URL?

Caching: a crawler that fetched robots.txt before you added the trap can lawfully use that copy for up to 24 hours under RFC 9309.

How does Cloudflare AI Labyrinth catch bots that ignore no-crawl directives?

If you're on Cloudflare, AI Labyrinth (announced March 19, 2025, opt-in on all plans including Free) does an automated version: when Cloudflare detects inappropriate bot activity it links in pre-generated decoy pages, shown only to suspected AI scrapers, marked so search engines don't index them, and treats bots that follow them deep as high-confidence automation for its detection models.

How do I ask Rightshold about this (free, no key)?

Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “This request hit my robots.txt-disallowed trap URL and carried Web Bot Auth headers. Does the signature verify for my site, and which signer does it belong to?”

One-click setup

{
  "tool": "gate_check",
  "arguments": {
    "policy": {
      "domain": "www.example.com",
      "require_web_bot_auth": true
    },
    "request": {
      "method": "GET",
      "url": "https://www.example.com/<trap path>",
      "headers": {
        "user-agent": "<as received>",
        "signature-agent": "<as received>",
        "signature-input": "<as received>",
        "signature": "<as received>"
      }
    }
  }
}

Rightshold doesn't host traps, read your logs, check IP lists or reverse DNS, or block traffic. gate_check only verifies a Web Bot Auth signature and policy you send it. A trap hit is evidence of what one client did on your site, not proof of who operates it or of any legal violation. Not legal advice.

What are the sources?

Related

Not legal advice. JSON · Markdown · All answers

Get a free API key in one call

No card and no email needed. 100 free credits; the license and policy checks stay free without a key.

curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_...   # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
  -H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'
Get a keyQuickstart