Answers · AI crawlers
How do I block spoofed ChatGPT-User, Perplexity-User and Googlebot requests at my edge without blocking the real ones?
Last updated
TL;DR: Make the edge rule check two things at once: the user-agent token and that the source IP is on that operator's own published list for that token (or passes a reverse-then-forward DNS check, or carries a verified Web Bot Auth signature). Anything that claims a bot name and fails the check is unverified automation, not that bot. OpenAI publishes chatgpt-user.json, searchbot.json and gptbot.json; Perplexity publishes perplexity-user.json and perplexitybot.json; Google publishes separate lists for common crawlers, special crawlers, user-triggered fetchers and user-triggered agents; Bing documents a search.msn.com DNS check. Load the lists into WAF IP lists that refresh on a schedule, write one rule per token, put the verify-and-skip rules above your block and challenge rules, and challenge or refuse the rest. This catches borrowed names, not bots that never claim one: Cloudflare reported in August 2025 that Perplexity, when blocked, also crawled with a generic Chrome user agent from IPs outside its published ranges, and only behavior-based bot detection caught it.
What should I check, step by step?
1. Pull each list from the operator: openai.com/chatgpt-user.json, searchbot.json and gptbot.json; www.perplexity.ai/perplexity-user.json and perplexitybot.json; Google's common-crawlers.json, special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json. They share one shape: a creationTime and a prefixes array of ipv4Prefix or ipv6Prefix CIDR ranges.
- Refresh on a schedule (daily is a reasonable default), alert when a fetch fails or creationTime stops moving, and keep the last good copy rather than loading an empty list. Perplexity's docs say its ranges are updated regularly and recommend automating the refresh.
- Pair each token with its own list: http.user_agent contains "Perplexity-User" and ip.src in $perplexity_user. A ChatGPT-User string from a GPTBot range, a residential IP or a random cloud host isn't ChatGPT-User. Perplexity's docs give this Cloudflare expression and the AWS WAF equivalent (an IP set match AND a User-Agent string match).
- For Googlebot and Bingbot, reverse-then-forward DNS works without a list: the hostname must end in googlebot.com, google.com or googleusercontent.com for Google, or search.msn.com for Bing, and must resolve back to the same IP. Cache the result per IP instead of doing two lookups on every request.
- Load IPv6 ranges as well as IPv4. Google's lists include IPv6 prefixes, and a rule built only from IPv4 ranges will treat the real crawler as a spoofer.
- Order the rules: verify-and-skip first, then block and challenge rules, because a Skip rule only bypasses rules that run after it. On Cloudflare, Perplexity's docs warn that Bot Fight Mode can't be skipped by custom rules and can still challenge a verified bot.
- Decide what a mismatch gets: a managed challenge or 403 on sensitive or paid paths, your normal anonymous-bot rate limits elsewhere. Where the operator signs requests, accept a verified Web Bot Auth signature as proof whatever IP it came from, and use the IP check as the fallback.
- Log the claimed token, which check passed (ip_list, rdns, signature) or mismatch, and the action taken. Review mismatches weekly: a burst of one bot name from one cloud network is usually a scraper wearing that name.
Why can't I trust a ChatGPT-User or Perplexity-User user-agent string on its own?
A user-agent is a header the client writes for itself. Google's verification guide exists for the case where spammers claim to be Google, and Perplexity's docs tell WAF users to combine user-agent matching with IP verification. Each operator pairs its tokens with published ranges for exactly this reason.
- Exemptions make spoofing pay. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user started the action, and Perplexity says Perplexity-User generally ignores robots.txt. If your rules wave those names through, anyone who copies the string gets the same pass, so the IP or signature check is the whole control.
How do I keep AI bot IP allowlists in my WAF up to date?
The lists move. On November 7, 2026, OpenAI's chatgpt-user.json had a creationTime of November 6, 2026 and Perplexity's perplexity-user.json one of October 28, 2026, while Bing's bingbot.json still showed January 3, 2024. For Bingbot, treat the DNS check that Bing documents as primary and the list as a second signal.
- On Cloudflare paid plans, Perplexity's docs suggest storing each bot's ranges in an IP list referenced from the rule (ip.src in $perplexitybot), so a refresh updates the list rather than the rule; free plans allow only one custom list. Cloudflare's own verified-bot fields, such as cf.verified_bot_category, are another option: Cloudflare says a verified bot identifies itself through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and doesn't abuse that access.
- Since July 1, 2026, Cloudflare also labels each verified bot or agent Direct (one operator) or Intermediary (a service many end users drive). For an intermediary you are trusting the operator, not every user behind it, so keep your per-agent limits on.
What spoofed or stealth AI crawler traffic can IP and user-agent checks miss?
Cloudflare's August 4, 2025 post reported that when sites blocked Perplexity's declared crawlers, Perplexity also fetched with a generic Chrome-on-macOS user agent from IPs outside its published ranges, rotating networks; Cloudflare counted 20 to 25 million daily requests from the declared Perplexity-User agent and 3 to 6 million from the undeclared one. Cloudflare said it removed Perplexity from its verified bots, caught the stealth traffic with machine learning and network signals, and that it couldn't pass managed challenges. In the same test, ChatGPT-User fetched robots.txt and stopped when disallowed.
- So matched names and IPs tell you which bots are honest about who they are. Traffic that presents as an ordinary browser needs bot scoring or a challenge, and agents that want a better lane than anonymous traffic can sign their requests or present a credential you can check.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “This request claims to come from ChatGPT and carries Signature headers, but I can't match its IP. Does its Web Bot Auth signature verify as chatgpt.com, so I can let it through on the signature instead of treating it as spoofed?”
{
"tool": "gate_check",
"arguments": {
"policy": {
"allowed_signature_agents": [
"chatgpt.com"
]
},
"request": {
"method": "GET",
"url": "<the requested URL>",
"headers": {
"user-agent": "<as received>",
"signature-agent": "<as received>",
"signature-input": "<as received>",
"signature": "<as received>"
}
}
}
}- Free, no key. It resolves the signer's HTTPS key directory, checks the signature against your policy and returns allow or deny with reasons, plus the signer it resolved. Allow answers can be cached for at most 5 minutes, and for less when the signature expires sooner.
- It verifies signatures, not addresses: it doesn't look up IP lists or reverse DNS. Keep the published-list rule at your edge for unsigned fetchers such as Perplexity-User and Googlebot.
- The Gate page (/gate) has free CC0 middleware snippets that run the same check in your stack, so the signature check can sit right after your IP rule fails.
Rightshold checks Web Bot Auth signatures and gate policies you send it. It doesn't sit in front of your site, maintain operators' IP lists, run DNS checks or block traffic, and it can't identify requests that carry no signature. Not legal advice.
What are the sources?
- OpenAI: Overview of OpenAI crawlers (published IP lists)
- Perplexity: Perplexity crawlers (IP lists, Cloudflare and AWS WAF rules)
- Google: Verify requests from Google crawlers and fetchers
- Bing Webmaster Tools: How to verify Bingbot
- Bing: bingbot.json IP list
- Cloudflare docs: Verified bots (Direct and Intermediary labels)
- Cloudflare docs: cf.verified_bot_category field
- Cloudflare blog: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (August 4, 2025)
- Cloudflare docs: Web Bot Auth
- The Gate on Rightshold
Related
- How to tell ChatGPT-User and other AI assistant fetches from AI training crawlers in server logs
- How to verify which AI agent is hitting your site with Web Bot Auth
- How to set per-agent rate limits and quotas that do not throttle human browsers
- Can you fingerprint an AI agent?
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'