# How do I tell ChatGPT-User and other AI assistant fetches apart from AI training crawlers in my server logs?

**Short answer:** Sort each AI request by the job its user-agent token names, then check the source IP against the operator's published list, because a user-agent string is just text anyone can send. The big operators split their bots by purpose. OpenAI: GPTBot crawls for model training, OAI-SearchBot crawls for ChatGPT search, and ChatGPT-User fetches a page when a person asks ChatGPT something; OpenAI says ChatGPT-User doesn't crawl automatically and, because a user started the action, robots.txt rules may not apply to it. Anthropic: ClaudeBot (training), Claude-SearchBot (search) and Claude-User (fetches at a user's request). Perplexity: PerplexityBot (search, not training) and Perplexity-User, which Perplexity says generally ignores robots.txt since a user asked. Google: Googlebot crawls, Google-Extended is only a robots.txt token with no user agent of its own, and Google-Agent is used by agents acting on a user's request. A ChatGPT-User or Claude-User hit usually means someone's assistant read that page for a question just now, which is the closest thing to a human visit these logs show.

## Checklist
1. Log enough to decide: the full user-agent, source IP, timestamp, path and status code. If your host or CDN strips query strings or user-agents from logs, fix that first.
2. Bucket by token, not by vendor: training (GPTBot, ClaudeBot, CCBot, Applebot-Extended and Google-Extended as robots.txt controls), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot), and user-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent).
3. Verify the IP for each claimed bot: OpenAI publishes separate lists (openai.com/gptbot.json, searchbot.json, chatgpt-user.json), Perplexity publishes perplexitybot.json and perplexity-user.json, Anthropic links a bots.json list from its crawler help article, and Google publishes common-crawlers, user-triggered-fetchers and user-triggered-agents JSON files.
4. For Google, the reverse-then-forward DNS check also works: the IP's hostname must end in googlebot.com, google.com or googleusercontent.com, and that hostname must resolve back to the same IP.
5. Match each token against its own IP list. A ChatGPT-User string from a GPTBot range, or any AI token from a residential or random cloud IP, is mislabeled or spoofed; count it as unknown.
6. Read the pattern: crawlers walk many URLs, fetch robots.txt and sitemaps, and come back on a schedule. User fetches hit one or a few pages, often seconds after someone could have asked about them, and often don't load your CSS, images or scripts.
7. Expect signed agents: ChatGPT's cloud browser signs requests with HTTP Message Signatures (RFC 9421), sending Signature, Signature-Input and Signature-Agent: "https://chatgpt.com"; verify against the keys at chatgpt.com/.well-known/http-message-signatures-directory, or let a CDN that checks them do it. Google says it is experimenting with Web Bot Auth for Google-Agent.
8. Decide policy per bucket in robots.txt (training and search tokens honor it), and check each user-triggered fetcher's own rule: Anthropic says disallowing Claude-User stops those fetches, while OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity and Google say their user-triggered fetchers generally ignore it, so blocking those takes a firewall rule.

## Why the buckets matter
- They answer different questions. Training crawls decide whether your content may end up in future models; search crawls decide whether you can be cited in AI search; user fetches are a person's assistant reading you right now. Blocking GPTBot doesn't remove you from ChatGPT search, and allowing OAI-SearchBot doesn't let GPTBot train on you; OpenAI says each setting is independent.
- User fetches are the ones worth counting as audience. A site with zero human page views can still have people reading it through ChatGPT, Claude or Perplexity, and those requests are the only trace in your logs.

## Common mistakes
- Trusting the user-agent alone. Scrapers copy GPTBot and Googlebot strings to get past rules; only the IP list, DNS check or a verified signature confirms it.
- Blocking by IP to opt out of training. Anthropic's help article says IP blocking may not work reliably as an opt-out because it stops its bots from reading your robots.txt; use robots.txt for that.
- Treating every Google request the same. Googlebot crawls for Search (including AI Overviews and AI Mode), Google-Extended is only a robots.txt token for Gemini training and grounding, and user-triggered fetchers such as Google-Agent generally ignore robots.txt.
- Forgetting robots.txt fetches. OpenAI may add a robots.txt marker to the GPTBot and OAI-SearchBot user-agent when it fetches robots.txt, which helps when your logs don't record paths.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "What does my site's robots.txt tell GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User and the other AI crawlers? Is AI training opted out?"

```json
{
  "tool": "read_ai_policy",
  "arguments": {
    "domain": "example.com"
  }
}
```
- Free, no key, cached 6 hours. It reads robots.txt per AI user agent (training, search and user-fetch tokens, including Content-Signal lines), ai.txt, /.well-known/tdmrep.json and whether llms.txt exists, then summarizes what each bot is told.
- It reads your published signals only. It doesn't read your server logs, verify IPs or signatures, or block anything; the log checks above are yours to run.

_Rightshold reads a site's public AI and crawl signals on request. It does not see your traffic, classify your logs or enforce robots.txt. Not legal advice._

## Sources
- [OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, published IP lists)](https://platform.openai.com/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [Google: Verify requests from Google crawlers and fetchers](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot)
- [Google: User-triggered fetchers (Google-Agent)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)
- [Google: Common crawlers (Google-Extended)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [OpenAI Help: ChatGPT cloud browser allowlisting (Web Bot Auth)](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting)
- [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421.html)

## Related
- [How to make a website readable to AI agents with llms.txt and markdown pages](https://www.rightshold.com/answers/how-to-make-a-website-readable-to-ai-agents-with-llms-txt-and-markdown)
- [Does a no-AI notice on your art do anything legally?](https://www.rightshold.com/answers/does-a-no-ai-notice-on-your-art-do-anything-legally)
- [Why are my one-time links already used? (link preview scanners)](https://www.rightshold.com/answers/why-are-my-one-time-links-already-used-link-preview-scanners)
- [How to verify an AI agent before doing business with it](https://www.rightshold.com/answers/how-to-verify-an-ai-agent-before-doing-business-with-it)

_Not legal advice. Updated 2026-10-30. JSON: https://www.rightshold.com/answers/how-to-tell-chatgpt-user-fetches-from-ai-training-crawlers-in-server-logs.json_