Answers · AI crawlers
Why does Google Analytics show almost no AI bot traffic, and how do I measure how much of my site's traffic is AI agents vs humans?
Last updated
TL;DR: Google Analytics can't see most AI traffic, so it can't tell you the split. GA4 counts visits when its JavaScript tag runs in a browser, and Google says it automatically removes traffic from known bots and spiders (using its own research and the IAB's International Spiders and Bots List). You can't turn that off or see how much it removed. Most AI crawlers never run the tag anyway: Vercel's December 2024 study found that none of the major AI crawlers it measured render JavaScript, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. What GA4 does show is people who click through from AI assistants: since May 13, 2026 it has an AI Assistant channel for visits from sources like ChatGPT, Gemini, Deepseek, Copilot or Grok, and ChatGPT adds utm_source=chatgpt.com to the links it shows. To measure the actual split, count from your server or CDN logs, where every request appears, and sort each one into verified crawler, AI user fetch, signed agent, unverified automation or likely human.
What should I check, step by step?
1. Use server or CDN logs as the denominator, not GA4. Keep user agent, source IP, path, status, time and any Signature-Agent header for at least a few weeks.
- Count page requests only. Drop static assets, health checks, your own monitors and deploy probes first, or they swamp the numbers.
- Sort each request into one bucket: verified search or AI crawler (IP in the operator's published list or a reverse-plus-forward DNS match), AI user-triggered fetch (ChatGPT-User, Claude-User, Perplexity-User, also verified by IP), signed agent (a Web Bot Auth signature that verifies), unverified automation (scripts, headless browsers, bot names from the wrong IPs) and likely human.
- Be strict about 'likely human'. A browser user agent isn't enough: also require a non-datacenter network, page assets loaded, and ideally a first-party beacon that only fires when JavaScript runs. Swarms of cloud IPs sending real browser user agents are common and will otherwise look like people.
- Report the split two ways, by requests and by unique visitors per day (IP plus user agent). Crawlers dominate request counts; humans look larger when you count visitors.
- Track AI click-throughs separately: in GA4, check the AI Assistant row in Traffic acquisition, then switch the dimension to Session source / medium and search for chatgpt, perplexity, claude, gemini and copilot. Google's AI Overviews and AI Mode clicks are counted under Organic Search, not AI Assistant.
- Compare the two sources every week. Log-based likely-human page views should be in the same range as GA4 views. If logs show many times more 'humans' than GA4, your human filter is letting automation through.
Why doesn't Google Analytics 4 show AI crawler and AI agent traffic?
Two filters stack up. First, the GA4 tag is JavaScript, so a client that downloads HTML without running scripts never sends a hit. Vercel's crawler study found ChatGPT's and Claude's crawlers do fetch JavaScript files (11.50% and 23.84% of their requests on nextjs.org) but don't execute them; Gemini, using Googlebot's infrastructure, and Applebot do render.
- Second, Google's help page says traffic from known bots and spiders is excluded automatically, and that at this time you can't disable that exclusion or see how much was excluded. So even a bot that runs your tag usually disappears.
- That's the right behaviour for marketing analytics and the wrong tool for capacity planning, bot policy or deciding whether AI agents matter to your business. Those questions need every request, which only your logs have.
What can Google Analytics 4 show about visits from ChatGPT and other AI assistants?
AI Assistant channel: Google's default channel definitions say the medium is set to ai-assistant and the campaign to (ai-assistant) when the referrer matches Google's list of AI assistants. The channel covers sources like ChatGPT, Gemini, Deepseek, Copilot or Grok and excludes Google's AI Overviews and AI Mode, which stay in Organic Search.
- ChatGPT tagging: OpenAI's publisher FAQ says ChatGPT automatically includes utm_source=chatgpt.com in referral URLs. Look for that source when a referrer is missing, and check that the visit landed in the channel you expect.
- These are people who clicked. They don't count the far larger number of times an assistant read your page for someone without a click; in logs that shows up as ChatGPT-User, Claude-User or Perplexity-User requests.
How do I calculate the share of my site traffic that is AI agents vs humans from server logs?
Say one day has 20,000 page requests after removing assets and your own probes. Verified crawlers (Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and others in their published ranges) are 9,000; AI user fetches 40; signed agents 10; unverified automation 10,500; likely humans 450. Automated share by requests is 97.75%.
- Count unique visitors the same way and humans will be a bigger share. Publish both numbers, the date range and your rules, so the split can be repeated next month with the same method.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “This request in my logs says it's an AI agent and carries Web Bot Auth headers. Does the signature verify for my site, so I can count it as a signed agent?”
{
"tool": "gate_check",
"arguments": {
"policy": {
"domain": "www.example.com",
"require_web_bot_auth": true
},
"request": {
"method": "GET",
"url": "https://www.example.com/<path>",
"headers": {
"user-agent": "<as received>",
"signature-agent": "<as received>",
"signature-input": "<as received>",
"signature": "<as received>"
}
}
}
}- Free, no key. If the signature verifies, the answer names the signer (the Signature-Agent host or URL), so the request goes in the signed-agent bucket under a real operator, not just a claimed name.
- A missing or failing signature is denied under require_web_bot_auth with the reason. Most crawlers don't sign yet, so check unsigned ones against the operator's IP list or reverse DNS instead.
Rightshold doesn't read your analytics or logs, count your traffic or block anything. gate_check verifies one Web Bot Auth signature against the policy you send. The bucket rules and example numbers above are a method, not a measurement of your site.
What are the sources?
- Google Analytics Help: Known bot-traffic exclusion (automatic; IAB list; can't disable or see how much)
- Google Analytics Help: Default channel group (AI Assistant channel; medium ai-assistant; AI Overviews and AI Mode in Organic Search)
- Google Analytics Help: What's new in Google Analytics (May 13, 2026: new AI Assistant traffic measurement)
- Vercel: The rise of the AI crawler (Dec 17, 2024; major AI crawlers don't render JavaScript)
- OpenAI Help Center: Publishers and Developers FAQ (utm_source=chatgpt.com in referral URLs)
- OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User; published IP lists)
- The Gate on Rightshold
Related
- How to tell ChatGPT-User fetches from AI training crawlers in server logs
- How to verify which AI agent is hitting your site with Web Bot Auth
- How to block spoofed ChatGPT-User and Perplexity-User requests at the edge without blocking the real ones
- How to catch crawlers that ignore robots.txt with a disallowed trap URL
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'