Answers · AI crawlers
How do I tell ChatGPT-User and other AI assistant fetches apart from AI training crawlers in my server logs?
Last updated
TL;DR: Sort each AI request by the job its user-agent token names, then check the source IP against the operator's published list, because a user-agent string is just text anyone can send. The big operators split their bots by purpose. OpenAI: GPTBot crawls for model training, OAI-SearchBot crawls for ChatGPT search, and ChatGPT-User fetches a page when a person asks ChatGPT something; OpenAI says ChatGPT-User doesn't crawl automatically and, because a user started the action, robots.txt rules may not apply to it. Anthropic: ClaudeBot (training), Claude-SearchBot (search) and Claude-User (fetches at a user's request). Perplexity: PerplexityBot (search, not training) and Perplexity-User, which Perplexity says generally ignores robots.txt since a user asked. Google: Googlebot crawls, Google-Extended is only a robots.txt token with no user agent of its own, and Google-Agent is used by agents acting on a user's request. A ChatGPT-User or Claude-User hit usually means someone's assistant read that page for a question just now, which is the closest thing to a human visit these logs show.
What should I check, step by step?
1. Log enough to decide: the full user-agent, source IP, timestamp, path and status code. If your host or CDN strips query strings or user-agents from logs, fix that first.
- Bucket by token, not by vendor: training (GPTBot, ClaudeBot, CCBot, Applebot-Extended and Google-Extended as robots.txt controls), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot), and user-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent).
- Verify the IP for each claimed bot: OpenAI publishes separate lists (openai.com/gptbot.json, searchbot.json, chatgpt-user.json), Perplexity publishes perplexitybot.json and perplexity-user.json, Anthropic links a bots.json list from its crawler help article, and Google publishes common-crawlers, user-triggered-fetchers and user-triggered-agents JSON files.
- For Google, the reverse-then-forward DNS check also works: the IP's hostname must end in googlebot.com, google.com or googleusercontent.com, and that hostname must resolve back to the same IP.
- Match each token against its own IP list. A ChatGPT-User string from a GPTBot range, or any AI token from a residential or random cloud IP, is mislabeled or spoofed; count it as unknown.
- Read the pattern: crawlers walk many URLs, fetch robots.txt and sitemaps, and come back on a schedule. User fetches hit one or a few pages, often seconds after someone could have asked about them, and often don't load your CSS, images or scripts.
- Expect signed agents: ChatGPT's cloud browser signs requests with HTTP Message Signatures (RFC 9421), sending Signature, Signature-Input and Signature-Agent: "https://chatgpt.com"; verify against the keys at chatgpt.com/.well-known/http-message-signatures-directory, or let a CDN that checks them do it. Google says it is experimenting with Web Bot Auth for Google-Agent.
- Decide policy per bucket in robots.txt (training and search tokens honor it), and check each user-triggered fetcher's own rule: Anthropic says disallowing Claude-User stops those fetches, while OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity and Google say their user-triggered fetchers generally ignore it, so blocking those takes a firewall rule.
Why the buckets matter
They answer different questions. Training crawls decide whether your content may end up in future models; search crawls decide whether you can be cited in AI search; user fetches are a person's assistant reading you right now. Blocking GPTBot doesn't remove you from ChatGPT search, and allowing OAI-SearchBot doesn't let GPTBot train on you; OpenAI says each setting is independent.
- User fetches are the ones worth counting as audience. A site with zero human page views can still have people reading it through ChatGPT, Claude or Perplexity, and those requests are the only trace in your logs.
Common mistakes
Trusting the user-agent alone. Scrapers copy GPTBot and Googlebot strings to get past rules; only the IP list, DNS check or a verified signature confirms it.
- Blocking by IP to opt out of training. Anthropic's help article says IP blocking may not work reliably as an opt-out because it stops its bots from reading your robots.txt; use robots.txt for that.
- Treating every Google request the same. Googlebot crawls for Search (including AI Overviews and AI Mode), Google-Extended is only a robots.txt token for Gemini training and grounding, and user-triggered fetchers such as Google-Agent generally ignore robots.txt.
- Forgetting robots.txt fetches. OpenAI may add a robots.txt marker to the GPTBot and OAI-SearchBot user-agent when it fetches robots.txt, which helps when your logs don't record paths.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “What does my site's robots.txt tell GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User and the other AI crawlers? Is AI training opted out?”
{
"tool": "read_ai_policy",
"arguments": {
"domain": "example.com"
}
}- Free, no key, cached 6 hours. It reads robots.txt per AI user agent (training, search and user-fetch tokens, including Content-Signal lines), ai.txt, /.well-known/tdmrep.json and whether llms.txt exists, then summarizes what each bot is told.
- It reads your published signals only. It doesn't read your server logs, verify IPs or signatures, or block anything; the log checks above are yours to run.
Rightshold reads a site's public AI and crawl signals on request. It does not see your traffic, classify your logs or enforce robots.txt. Not legal advice.
What are the sources?
- OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, published IP lists)
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google: Verify requests from Google crawlers and fetchers
- Google: User-triggered fetchers (Google-Agent)
- Google: Common crawlers (Google-Extended)
- OpenAI Help: ChatGPT cloud browser allowlisting (Web Bot Auth)
- RFC 9421: HTTP Message Signatures
Related
- How to make a website readable to AI agents with llms.txt and markdown pages
- Does a no-AI notice on your art do anything legally?
- Why are my one-time links already used? (link preview scanners)
- How to verify an AI agent before doing business with it
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'