{
  "question": "How do I catch AI crawlers and bots that ignore robots.txt, using a disallowed trap URL?",
  "slug": "how-to-catch-crawlers-that-ignore-robots-txt-with-a-disallowed-trap-url",
  "url": "https://www.rightshold.com/answers/how-to-catch-crawlers-that-ignore-robots-txt-with-a-disallowed-trap-url",
  "short_answer": "Add a path to robots.txt as Disallow for every user agent, never link to it or put it in your sitemap, give it a name no one would guess, and log every request to it. A client that fetches that path almost certainly read your robots.txt and went where it was told not to; nothing else points there. A second trap catches crawlers that never read robots.txt: a disallowed URL that is linked from pages, so only link-followers that skip the robots.txt check reach it. Before you call a hit a violation, rule out three innocent cases. RFC 9309 lets crawlers cache robots.txt for up to 24 hours (OpenAI says its search systems can take about 24 hours to adjust to a change), so wait a day after adding the trap. User-triggered fetchers aren't bound the same way: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. And RFC 9309 notes that listing a path in robots.txt makes it discoverable, so security scanners and curious people will hit it too. Then identify the visitor: check the claimed bot's published IP list or reverse DNS, or verify a Web Bot Auth signature if the request carried one, which gate_check does for free.",
  "checklist": [
    "Pick a trap path that looks plausible but is unique and unguessable (a random segment), add Disallow: /that-path/ under User-agent: *, and keep it out of the sitemap and all links.",
    "Make sure robots.txt returns 200. Under RFC 9309 a 4xx robots.txt lets crawlers access anything and a 5xx means they must assume complete disallow, so a broken file changes what counts as a violation.",
    "Serve the trap a plain 404 or an empty noindex page. Don't serve content worth taking, and don't redirect.",
    "Log the full request on a hit: time, IP, user agent, every header (including any Signature, Signature-Input and Signature-Agent) and whether the same IP fetched robots.txt shortly before.",
    "Wait at least 24 hours after publishing the trap before treating hits as violations; RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours, and Google says it generally caches it for up to 24 hours.",
    "Classify each hit: verified bot (IP list, reverse DNS or a valid signature), user-triggered fetcher (ChatGPT-User, Perplexity-User and similar, which say robots.txt may not apply), scanner or human, or unverified automation claiming a bot name.",
    "Optionally add a second, linked trap: a normal-looking link to another disallowed path. Keep it a real link a person could see; Google's spam policies treat links hidden with CSS, zero-size text or off-screen positioning as hidden link abuse when used to manipulate search.",
    "Act on verified identities only. Block or rate-limit a verified operator that keeps hitting the trap; for unverified hits, block the pattern at your edge and don't blame the bot whose name was borrowed."
  ],
  "sections": [
    {
      "heading": "Why a trap is evidence and a user-agent string isn't",
      "points": [
        "robots.txt is a request, not a lock: RFC 9309 says its rules are not a form of access authorization. Your server logs show who asked for robots.txt and which paths each client then fetched, but anyone can send any user-agent string, so a hit only tells you what a client claimed to be.",
        "A disallowed, unlinked path closes most innocent explanations. A search engine can't have found it in a link and a person can't have stumbled on it; the only public pointer is the Disallow line. OpenAI says some of its fetches of robots.txt carry an extra robots.txt marker in the user agent, which makes the robots.txt read easy to pair with what came after.",
        "Cloudflare used the same idea at scale. In its August 4, 2025 post it describes buying brand-new domains, disallowing all crawling in robots.txt and blocking Perplexity's declared crawlers, then finding Perplexity could answer questions about content on those domains. It reported a declared Perplexity-User agent and an undeclared generic Chrome-on-macOS agent, the latter from IPs outside Perplexity's published ranges."
      ]
    },
    {
      "heading": "The innocent explanations to rule out first",
      "points": [
        "Caching: a crawler that fetched robots.txt before you added the trap can lawfully use that copy for up to 24 hours under RFC 9309.",
        "User-triggered fetches: OpenAI states that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. A hit from a verified user-triggered fetcher usually means a person's assistant went there, which you may still choose to block, but it isn't a crawler breaking its stated policy.",
        "Discovery: RFC 9309's security section says listing paths in robots.txt makes them discoverable. Vulnerability scanners read robots.txt precisely to find such paths, so scanner hits are expected and say nothing about AI crawlers."
      ]
    },
    {
      "heading": "Going further: Cloudflare AI Labyrinth",
      "points": [
        "If you're on Cloudflare, AI Labyrinth (announced March 19, 2025, opt-in on all plans including Free) does an automated version: when Cloudflare detects inappropriate bot activity it links in pre-generated decoy pages, shown only to suspected AI scrapers, marked so search engines don't index them, and treats bots that follow them deep as high-confidence automation for its detection models.",
        "Cloudflare's own post notes the limit of simple traps: bots now look for honeypot techniques like hidden links, which is why it moved to networks of realistic decoy pages. A hand-made trap still catches careless crawlers cheaply; it won't catch one that checks robots.txt before every fetch and then disguises itself elsewhere."
      ]
    }
  ],
  "checks": [],
  "ask_rightshold": {
    "start": "https://www.rightshold.com/start?ref=seo",
    "mcp_endpoint": "https://www.rightshold.com/mcp",
    "ask": "This request hit my robots.txt-disallowed trap URL and carried Web Bot Auth headers. Does the signature verify for my site, and which signer does it belong to?",
    "mcp_call": {
      "tool": "gate_check",
      "arguments": {
        "policy": {
          "domain": "www.example.com",
          "require_web_bot_auth": true
        },
        "request": {
          "method": "GET",
          "url": "https://www.example.com/<trap path>",
          "headers": {
            "user-agent": "<as received>",
            "signature-agent": "<as received>",
            "signature-input": "<as received>",
            "signature": "<as received>"
          }
        }
      }
    },
    "also": [
      "Free, no key. If the signature verifies, the answer names the signer (the Signature-Agent host or URL), so you know which operator to block or contact, not just which name the request claimed.",
      "A request with no signature, or one that doesn't verify, is denied under require_web_bot_auth with the reason. That doesn't make it malicious; check the claimed bot's published IP list or reverse DNS next.",
      "Most AI crawlers don't sign requests yet, so expect many trap hits to have no signature at all."
    ],
    "scope": "Rightshold doesn't host traps, read your logs, check IP lists or reverse DNS, or block traffic. gate_check only verifies a Web Bot Auth signature and policy you send it. A trap hit is evidence of what one client did on your site, not proof of who operates it or of any legal violation. Not legal advice."
  },
  "sources": [
    {
      "title": "RFC 9309: Robots Exclusion Protocol (not access authorization; 4xx/5xx handling; 24-hour cache; listed paths become discoverable)",
      "url": "https://www.rfc-editor.org/rfc/rfc9309.html"
    },
    {
      "title": "Google Search Central: How Google interprets the robots.txt specification (caching up to 24 hours)",
      "url": "https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt"
    },
    {
      "title": "Google Search Central: Spam policies (hidden text and link abuse)",
      "url": "https://developers.google.com/search/docs/essentials/spam-policies"
    },
    {
      "title": "OpenAI: Overview of OpenAI crawlers (ChatGPT-User and robots.txt, ~24 hours to adjust, robots.txt user-agent marker)",
      "url": "https://developers.openai.com/api/docs/bots"
    },
    {
      "title": "Perplexity: Perplexity crawlers (Perplexity-User generally ignores robots.txt)",
      "url": "https://docs.perplexity.ai/guides/bots"
    },
    {
      "title": "Cloudflare: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (Aug 4, 2025)",
      "url": "https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/"
    },
    {
      "title": "Cloudflare: Trapping misbehaving bots in an AI Labyrinth (Mar 19, 2025)",
      "url": "https://blog.cloudflare.com/ai-labyrinth/"
    },
    {
      "title": "The Gate on Rightshold",
      "url": "https://www.rightshold.com/gate"
    }
  ],
  "related": [
    {
      "title": "Why an AI-training opt-out or robots.txt is not access control (and what to use instead)",
      "url": "https://www.rightshold.com/answers/why-an-ai-training-opt-out-or-robots-txt-is-not-access-control"
    },
    {
      "title": "How to tell ChatGPT-User fetches from AI training crawlers in server logs",
      "url": "https://www.rightshold.com/answers/how-to-tell-chatgpt-user-fetches-from-ai-training-crawlers-in-server-logs"
    },
    {
      "title": "How to block spoofed ChatGPT-User and Perplexity-User requests at the edge without blocking the real ones",
      "url": "https://www.rightshold.com/answers/how-to-block-spoofed-chatgpt-user-and-perplexity-user-requests-at-the-edge-without-blocking-the-real-ones"
    },
    {
      "title": "How to block AI training crawlers in robots.txt without leaving AI search",
      "url": "https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search"
    }
  ],
  "computed_from": "Curated guidance; license verdicts computed from Rightshold's license profiles at render time. Not legal advice.",
  "updated": "2026-11-09"
}