{
  "question": "How do I detect AI agents and scrapers that send an ordinary Chrome user agent instead of declaring themselves?",
  "slug": "how-to-detect-ai-agents-that-pose-as-a-regular-chrome-browser",
  "url": "https://www.rightshold.com/answers/how-to-detect-ai-agents-that-pose-as-a-regular-chrome-browser",
  "short_answer": "You can't catch them from the user-agent string, because it is the part they copy. Score each request on signals that are harder to fake together: a TLS fingerprint (JA3/JA4) that doesn't match the browser the user-agent claims, missing or inconsistent fetch-metadata headers (Sec-Fetch-Mode, Sec-Fetch-Dest) on HTTPS page navigations, automation flags in the page (navigator.webdriver is true when Chrome runs with --enable-automation or --headless), and behaviour no person produces, such as HTML with no CSS or JS loads, many pages per second, or the same few URLs fetched from rotating cloud IPs. Combine them into a score and challenge on sensitive paths instead of blocking on one signal. Cloudflare reported on August 4, 2025 that Perplexity, when blocked under its declared names, also fetched with a generic Chrome-on-macOS user agent from IPs outside its published ranges, and that Cloudflare caught this with machine learning and network signals, not names. Give agents that want access a better lane: a verified Web Bot Auth signature. gate_check verifies those signatures for free; it can't classify unsigned traffic.",
  "checklist": [
    "Stop treating the user-agent as identity. A Chrome string from a cloud network tells you only what the client chose to send. Matching a declared bot name to its published IP list is a separate job (see the spoofed ChatGPT-User page); this page is about traffic that declares nothing.",
    "Compare the TLS fingerprint with the claimed browser. Cloudflare's docs describe JA3/JA4 as identifying TLS clients by how they start connections, so a script library or headless stack can carry a Chrome user-agent but a non-Chrome fingerprint. On Cloudflare the field is cf.bot_management.ja4 (Enterprise Bot Management only); elsewhere, FoxIO's open-source JA4 suite can compute it at your proxy.",
    "Check fetch metadata on page loads. A browser navigating to an HTTPS page sends Sec-Fetch-Mode: navigate and Sec-Fetch-Dest: document (MDN). A request that claims a current Chrome and arrives with none of these on a top-level page is worth scoring as automated. It is a signal, not proof: a proxy can strip headers and a script can add them.",
    "Read automation flags in the page if you run JavaScript checks. MDN documents that navigator.webdriver is true in Chrome when --enable-automation or --headless is used, or --remote-debugging-port is 0. Stealth tooling can patch this, so a false value doesn't clear anyone.",
    "Score behaviour from your own logs: page HTML without the stylesheet, script and image requests a browser makes; many distinct pages from one IP inside a second; the same handful of URLs fetched by many datacenter IPs; robots.txt never fetched; no cookies carried between pages.",
    "Combine signals into one score and act by path. Challenge (not block) on login, checkout, search and API routes; leave cacheable public pages lighter. On Cloudflare, bot scores run 1 to 99, where 1 means quite certainly automated; granular scores need Enterprise Bot Management, and Pro and higher plans see bot groupings.",
    "Offer a verified lane. Agents and remote-browser platforms that sign requests with Web Bot Auth can be told apart from anonymous automation; on Cloudflare, cf.bot_management.signed_agent marks a known agent that self-identifies with Web Bot Auth. Let verified agents skip the challenge on the paths you choose.",
    "Review weekly what you challenged and who complained. A rule that blocks real people on a corporate proxy (for example one that strips the user-agent) costs more than the scraper it stopped."
  ],
  "sections": [
    {
      "heading": "Why a matched name and IP doesn't cover this traffic",
      "points": [
        "Published IP lists and reverse-DNS checks answer one question: is this request that claims to be GPTBot or Googlebot really from that operator? They say nothing about a request that claims to be Chrome on Windows. Undeclared automation is designed to fall into your human bucket.",
        "Cloudflare's August 4, 2025 post described exactly that pattern. After sites blocked Perplexity's declared crawlers, Cloudflare saw fetches with a generic Chrome-on-macOS user agent from IPs outside Perplexity's published ranges, rotating across networks. It reported 20 to 25 million daily requests from the declared agent and 3 to 6 million from the undeclared one, said it fingerprinted the stealth traffic with machine learning and network signals, and said that traffic could not pass managed challenges."
      ]
    },
    {
      "heading": "Signals that are hard to fake all at once",
      "points": [
        "TLS: Cloudflare's JA3/JA4 docs say each client type has distinct connection characteristics, so the fingerprint is a stable identifier across destinations; JA4 sorts ClientHello extensions to reduce noise from modern browsers. Cloudflare's JA4 Signals add last-hour aggregates per fingerprint across its network, such as browser_ratio_1h (share of requests from browser user agents) and uas_rank_1h (how many distinct user agents share the fingerprint). A fingerprint seen with many different user agents is a classic sign of a tool rotating strings.",
        "Missing fingerprints are not evidence: Cloudflare notes JA3/JA4 are absent on plain HTTP, when Bot Management is skipped, and after TLS session resumption.",
        "HTTP: fetch-metadata headers are only sent to potentially trustworthy (HTTPS) URLs, and a top-level navigation carries navigate/document. Script clients often omit them or send combinations a browser wouldn't.",
        "In-page: Cloudflare's JavaScript detections engine targets headless browsers and other automation tools; it injects a lightweight script and either blocks, challenges or passes the request to other engines. navigator.webdriver is a cooperative flag for the same purpose.",
        "Behaviour: no single request looks wrong, but sessions do. HTML-only loads, perfectly even timing, no asset or favicon requests, and identical paths across many IPs add up quickly."
      ]
    },
    {
      "heading": "Challenge, score, and give signed agents a door",
      "points": [
        "Treat the score as a reason to ask for more, not a verdict. A managed challenge on sensitive paths stops most headless traffic while a person clicks through; hard blocks belong on clear abuse such as credential stuffing.",
        "Cloudflare notes requests with a missing or empty user-agent get bot score 1 from its heuristics, and that this is a common false positive behind Zero Trust or corporate proxies that strip the header. Exempt known corporate egress before you tighten rules.",
        "Agents you want can sign. Web Bot Auth uses HTTP Message Signatures with a key directory at /.well-known/http-message-signatures-directory, so a verified signature identifies the operator whatever IP or user-agent it uses. Put a skip-on-verified-signature rule above your challenge rules, and keep per-agent rate limits on."
      ]
    }
  ],
  "checks": [],
  "ask_rightshold": {
    "start": "https://www.rightshold.com/start?ref=seo",
    "mcp_endpoint": "https://www.rightshold.com/mcp",
    "ask": "This request says it's Chrome but my bot score says automated. It carries Signature headers. Does it verify as a Web Bot Auth agent, so I can let it skip the challenge instead of treating it as a disguised scraper?",
    "mcp_call": {
      "tool": "gate_check",
      "arguments": {
        "policy": {
          "require_web_bot_auth": true
        },
        "request": {
          "method": "GET",
          "url": "<the requested URL>",
          "headers": {
            "user-agent": "<as received>",
            "signature-agent": "<as received>",
            "signature-input": "<as received>",
            "signature": "<as received>"
          }
        }
      }
    },
    "also": [
      "Free, no key. It resolves the signer's HTTPS key directory, verifies the Web Bot Auth signature and returns allow or deny with reasons and the signer it resolved. Add allowed_signature_agents to accept only signers you name.",
      "It verifies signatures only. It doesn't read TLS fingerprints, bot scores or IP reputation, so it can't tell you whether an unsigned Chrome request is a person or a script; that stays with your edge or bot-management tool.",
      "The Gate page (/gate) has free CC0 middleware snippets that run the same check in your stack, so a verified agent can skip the challenge while unsigned traffic keeps your normal scoring."
    ],
    "scope": "Rightshold checks Web Bot Auth signatures and gate policies you send it. It doesn't sit in front of your site, compute JA3/JA4 fingerprints, run JavaScript challenges, score bots or block traffic, and it can't identify requests that carry no signature. Not legal advice."
  },
  "sources": [
    {
      "title": "Cloudflare: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (Aug 4, 2025)",
      "url": "https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/"
    },
    {
      "title": "Cloudflare docs: JA3/JA4 fingerprint and Signals Intelligence",
      "url": "https://developers.cloudflare.com/bots/additional-configurations/ja3-ja4-fingerprint/"
    },
    {
      "title": "FoxIO: JA4+ network fingerprinting (open source)",
      "url": "https://github.com/FoxIO-LLC/ja4"
    },
    {
      "title": "Cloudflare docs: Bot scores (1-99, detection engines, empty user-agent note)",
      "url": "https://developers.cloudflare.com/bots/concepts/bot-score/"
    },
    {
      "title": "Cloudflare docs: Bot Management variables (ja4, signed_agent, verified bot category)",
      "url": "https://developers.cloudflare.com/bots/reference/bot-management-variables/"
    },
    {
      "title": "Cloudflare docs: JavaScript detections",
      "url": "https://developers.cloudflare.com/bots/additional-configurations/javascript-detections/"
    },
    {
      "title": "MDN: Navigator.webdriver",
      "url": "https://developer.mozilla.org/en-US/docs/Web/API/Navigator/webdriver"
    },
    {
      "title": "MDN: Sec-Fetch-Mode header",
      "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Sec-Fetch-Mode"
    },
    {
      "title": "Cloudflare docs: Web Bot Auth",
      "url": "https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/"
    },
    {
      "title": "The Gate on Rightshold",
      "url": "https://www.rightshold.com/gate"
    }
  ],
  "related": [
    {
      "title": "How to block spoofed ChatGPT-User and Perplexity-User requests at the edge without blocking the real ones",
      "url": "https://www.rightshold.com/answers/how-to-block-spoofed-chatgpt-user-and-perplexity-user-requests-at-the-edge-without-blocking-the-real-ones"
    },
    {
      "title": "How to verify which AI agent is hitting your site with Web Bot Auth",
      "url": "https://www.rightshold.com/answers/how-to-verify-which-ai-agent-is-hitting-your-site-with-web-bot-auth"
    },
    {
      "title": "How to tell ChatGPT-User fetches from AI training crawlers in server logs",
      "url": "https://www.rightshold.com/answers/how-to-tell-chatgpt-user-fetches-from-ai-training-crawlers-in-server-logs"
    },
    {
      "title": "How to set per-agent rate limits and quotas that do not throttle human browsers",
      "url": "https://www.rightshold.com/answers/how-to-set-per-agent-rate-limits-and-quotas-that-do-not-throttle-human-browsers"
    }
  ],
  "computed_from": "Curated guidance; license verdicts computed from Rightshold's license profiles at render time. Not legal advice.",
  "updated": "2026-11-08"
}