Answers · AI crawlers
Requests with Google-Agent in the user agent are showing up in my logs. What is it, will a robots.txt rule stop it, and how do I check it's really Google and not someone borrowing the name?
Last updated
TL;DR: Google-Agent is the user agent Google lists for agents hosted on Google infrastructure that navigate the web and perform actions when a user asks. Google files it under user-triggered fetchers, and says that because a person requested the fetch, these fetchers generally ignore robots.txt, so a Google-Agent group in robots.txt isn't a dependable way to keep it out; a rule at your server or edge is. Googlebot rules don't cover it either, because it's a separate token. To check a request, match the Google-Agent token, then the source IP against user-triggered-agents.json, the list Google names for it (20 prefixes, 12 IPv4 and 8 IPv6, created November 13, 2026, when fetched November 14). Some requests also carry an experimental Web Bot Auth signature: Signature-Agent g="https://agent.bot.goog", with keys at agent.bot.goog/.well-known/http-message-signatures-directory. Google says it doesn't sign every request, so keep the IP check as the fallback.
What should I check, step by step?
1. Match the Google-Agent token anywhere in the User-Agent header. Google's desktop string puts it inside the browser's own parentheses and the mobile string puts it at the end, and the Chrome version is shown as W.X.Y.Z, so it changes.
- Don't count on robots.txt: Google says user-triggered fetchers generally ignore it because a user asked for the fetch. Decide Google-Agent with a server or edge rule.
- Don't expect your Googlebot or GoogleOther groups to apply. Google-Agent is its own token, and Google Search crawling stays with Googlebot.
- Check the source IP against https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json, the list Google names for Google-Agent, and refresh it on a schedule.
- If a request carries Signature-Agent: g="https://agent.bot.goog", verify the g-labeled Signature and Signature-Input headers (RFC 9421) against the keys in Google's directory, cached per its Cache-Control.
- Drop cached keys that disappear from Google's directory. Google says missing keys may be expired or revoked.
- Ask your CDN or bot tool how it handles Google-Agent before relying on a verified flag. Google signs with the dictionary form of Signature-Agent, which Cloudflare's own signing guide says it doesn't verify for bots that register with it.
- If you let verified Google-Agent traffic through, keep sign-in, payment and account changes behind whatever you already require of people.
What is Google-Agent in a user agent string?
Google's list of user-triggered fetchers (developers.google.com, read November 14, 2026) says Google-Agent "is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request" and that it uses IP ranges from user-triggered-agents.json. The same page says user-triggered fetchers are started by users to perform a fetching function within a Google product.
- Google publishes two strings. Mobile: "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)". Desktop: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36".
- Because it carries a full Chrome string, a quick look at your logs can file it as a person's visit. Search for the token itself, and keep it apart from Googlebot (Search), GoogleOther (product teams' research crawls) and Google-Extended (a robots.txt control token with no crawler of its own).
Does Google-Agent follow robots.txt?
Google's page says that because the fetch was requested by a user, its user-triggered fetchers "generally ignore robots.txt rules". That's the same position OpenAI takes for ChatGPT-User and Perplexity takes for Perplexity-User, and it's the opposite of Google's common crawlers such as GoogleOther, which Google says always obey robots.txt on automatic crawls.
- So a "User-agent: Google-Agent" group documents what you want but isn't an enforcement point. If you need to refuse it, do it where you can see the request: a firewall, CDN or server rule keyed on the token plus the IP list, answering 403 or a page that explains how to reach you.
- Refusing it doesn't change Google Search, which is crawled by Googlebot and governed by your Googlebot rules. What you give up is people using Google's hosted agents to read or act on your site for them.
How do you verify that a Google-Agent request really comes from Google?
IP list: fetched November 14, 2026, user-triggered-agents.json had a creationTime of 2026-11-13T15:45:59 and 20 prefixes: 12 IPv4 (in 136.121.x, 136.122.0.0/16 and 74.125.232.x) and 8 IPv6 (2001:4860:c::/124 blocks). Google also publishes user-triggered-fetchers.json and user-triggered-fetchers-google.json for its other fetchers, but names user-triggered-agents.json for Google-Agent. The reverse DNS patterns Google gives cover all user-triggered fetchers, so they don't single out Google-Agent; the IP list does.
- Signature: Google's Web Bot Auth guide (marked experimental) says a subset of Google-Agent requests are signed and authenticated as https://agent.bot.goog. Participating requests send Signature-Agent set to g="https://agent.bot.goog"; you fetch the key set from https://agent.bot.goog/.well-known/http-message-signatures-directory, cache it per Cache-Control, and verify the g-labeled Signature against Signature-Input under RFC 9421. On November 14, 2026 that directory held five Ed25519 keys, was served with Cache-Control max-age=3600, and was itself signed.
- Google says it isn't signing every request, not all of its user agents use the protocol, and you should keep relying on IP addresses, reverse DNS and user-agent strings during the rollout. A missing signature on a Google-Agent request from a listed address isn't a sign of a fake. A valid signature tells you who sent the request, not that the user behind it is allowed to do what they're asking.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “This request says it's Google-Agent and carries Signature headers. Does its Web Bot Auth signature verify as agent.bot.goog before I let it skip my bot challenge?”
{
"tool": "gate_check",
"arguments": {
"policy": {
"allowed_signature_agents": [
"agent.bot.goog"
]
},
"request": {
"method": "GET",
"url": "<the requested URL>",
"headers": {
"user-agent": "<as received>",
"signature-agent": "<as received>",
"signature-input": "<as received>",
"signature": "<as received>"
}
}
}
}- Free, no key. It resolves the signer's HTTPS key directory, verifies the Web Bot Auth signature against your policy and returns allow or deny with reasons, plus the signer it resolved. Allow answers can be cached for at most 5 minutes, and for less when the signature expires sooner.
- It verifies signatures only. It doesn't read user-triggered-agents.json, check IP addresses or reverse DNS, so keep the IP-list rule at your edge for the Google-Agent requests Google doesn't sign.
Rightshold checks agent signatures you pass it. It isn't affiliated with Google or Cloudflare, can't see Google-Agent traffic or your logs, and can't make any company honor a preference. User-agent strings, IP lists, key directory contents and robots.txt behavior are as Google published them, and Cloudflare's Signature-Agent format as its guide states, all read November 14, 2026; Google calls its Web Bot Auth use experimental and the lists and keys can change at any refresh. Not legal advice.
What are the sources?
- Google: List of Google's user-triggered fetchers (Google-Agent)
- Google: Authenticating requests with Web Bot Auth (experimental)
- Google: user-triggered-agents.json IP ranges
- Google-Agent key directory
- Cloudflare: Web Bot Auth (Signature-Agent format)
Related
- What is GoogleOther, and does blocking it affect Google Search?
- What is Claude-Agent, and how do you verify Claude Duet's browser visits to your site?
- How to verify which AI agent is hitting your site with Web Bot Auth
- How to block spoofed ChatGPT-User and Perplexity-User requests at the edge without blocking the real ones
- Can Google's buy for me agentic checkout buy without asking or charge more than you confirmed?
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'