How do I detect AI agents and scrapers that send an ordinary Chrome user agent instead of declaring themselves?

What should I check, step by step?

1. Stop treating the user-agent as identity. A Chrome string from a cloud network tells you only what the client chose to send. Matching a declared bot name to its published IP list is a separate job (see the spoofed ChatGPT-User page); this page is about traffic that declares nothing.

2. Compare the TLS fingerprint with the claimed browser. Cloudflare's docs describe JA3/JA4 as identifying TLS clients by how they start connections, so a script library or headless stack can carry a Chrome user-agent but a non-Chrome fingerprint. On Cloudflare the field is cf.bot_management.ja4 (Enterprise Bot Management only); elsewhere, FoxIO's open-source JA4 suite can compute it at your proxy.
3. Check fetch metadata on page loads. A browser navigating to an HTTPS page sends Sec-Fetch-Mode: navigate and Sec-Fetch-Dest: document (MDN). A request that claims a current Chrome and arrives with none of these on a top-level page is worth scoring as automated. It is a signal, not proof: a proxy can strip headers and a script can add them.
4. Read automation flags in the page if you run JavaScript checks. MDN documents that navigator.webdriver is true in Chrome when --enable-automation or --headless is used, or --remote-debugging-port is 0. Stealth tooling can patch this, so a false value doesn't clear anyone.
5. Score behaviour from your own logs: page HTML without the stylesheet, script and image requests a browser makes; many distinct pages from one IP inside a second; the same handful of URLs fetched by many datacenter IPs; robots.txt never fetched; no cookies carried between pages.
6. Combine signals into one score and act by path. Challenge (not block) on login, checkout, search and API routes; leave cacheable public pages lighter. On Cloudflare, bot scores run 1 to 99, where 1 means quite certainly automated; granular scores need Enterprise Bot Management, and Pro and higher plans see bot groupings.
7. Offer a verified lane. Agents and remote-browser platforms that sign requests with Web Bot Auth can be told apart from anonymous automation; on Cloudflare, cf.bot_management.signed_agent marks a known agent that self-identifies with Web Bot Auth. Let verified agents skip the challenge on the paths you choose.
8. Review weekly what you challenged and who complained. A rule that blocks real people on a corporate proxy (for example one that strips the user-agent) costs more than the scraper it stopped.

Why don't published IP lists catch AI agents that pretend to be Chrome?

Published IP lists and reverse-DNS checks answer one question: is this request that claims to be GPTBot or Googlebot really from that operator? They say nothing about a request that claims to be Chrome on Windows. Undeclared automation is designed to fall into your human bucket.

• Cloudflare's August 4, 2025 post described exactly that pattern. After sites blocked Perplexity's declared crawlers, Cloudflare saw fetches with a generic Chrome-on-macOS user agent from IPs outside Perplexity's published ranges, rotating across networks. It reported 20 to 25 million daily requests from the declared agent and 3 to 6 million from the undeclared one, said it fingerprinted the stealth traffic with machine learning and network signals, and said that traffic could not pass managed challenges.

Which signals reveal an AI agent or scraper sending a normal Chrome user agent?

TLS: Cloudflare's JA3/JA4 docs say each client type has distinct connection characteristics, so the fingerprint is a stable identifier across destinations; JA4 sorts ClientHello extensions to reduce noise from modern browsers. Cloudflare's JA4 Signals add last-hour aggregates per fingerprint across its network, such as browser_ratio_1h (share of requests from browser user agents) and uas_rank_1h (how many distinct user agents share the fingerprint). A fingerprint seen with many different user agents is a classic sign of a tool rotating strings.

• Missing fingerprints are not evidence: Cloudflare notes JA3/JA4 are absent on plain HTTP, when Bot Management is skipped, and after TLS session resumption.
• HTTP: fetch-metadata headers are only sent to potentially trustworthy (HTTPS) URLs, and a top-level navigation carries navigate/document. Script clients often omit them or send combinations a browser wouldn't.
• In-page: Cloudflare's JavaScript detections engine targets headless browsers and other automation tools; it injects a lightweight script and either blocks, challenges or passes the request to other engines. navigator.webdriver is a cooperative flag for the same purpose.
• Behaviour: no single request looks wrong, but sessions do. HTML-only loads, perfectly even timing, no asset or favicon requests, and identical paths across many IPs add up quickly.

Should I block or challenge suspected AI agents that pose as browsers?

Treat the score as a reason to ask for more, not a verdict. A managed challenge on sensitive paths stops most headless traffic while a person clicks through; hard blocks belong on clear abuse such as credential stuffing.

• Cloudflare notes requests with a missing or empty user-agent get bot score 1 from its heuristics, and that this is a common false positive behind Zero Trust or corporate proxies that strip the header. Exempt known corporate egress before you tighten rules.
• Agents you want can sign. Web Bot Auth uses HTTP Message Signatures with a key directory at /.well-known/http-message-signatures-directory, so a verified signature identifies the operator whatever IP or user-agent it uses. Put a skip-on-verified-signature rule above your challenge rules, and keep per-agent rate limits on.