Answers · AI crawlers
GoogleOther keeps crawling my site. What does Google use it for, will disallowing it in robots.txt hurt my Google Search rankings, and how do I tell the real one from a fake?
Last updated
TL;DR: GoogleOther is Google's generic crawler. Google's list of common crawlers (last updated October 30, 2026) says various product teams may use it to fetch publicly accessible content, for example one-off crawls for internal research and development, and that rules addressed to it don't affect any specific product. Google Search runs on Googlebot, and Google says Googlebot rules are the ones that affect Search and all its features, so a GoogleOther group doesn't touch your rankings. GoogleOther's only robots.txt token is GoogleOther; GoogleOther-Image and GoogleOther-Video answer to their own token and to GoogleOther, so one GoogleOther group covers all three unless you give them their own. With no GoogleOther group it follows your * group, never your Googlebot group. As a common crawler, Google says it always obeys robots.txt on automatic crawls, crawls from the ranges in common-crawlers.json and resolves to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com, so you can verify it like Googlebot.
What should I check, step by step?
1. Read the token: GoogleOther is Google's generic crawler for product teams' fetches, such as one-off research crawls. Google says rules for it don't affect any specific product.
- To refuse it, add "User-agent: GoogleOther" followed by "Disallow: /". Google Search keeps using Googlebot and your Googlebot rules.
- One group covers the family: GoogleOther-Image and GoogleOther-Video also answer to GoogleOther, so they follow a GoogleOther group unless you give them groups of their own.
- Don't count on your Googlebot group: GoogleOther doesn't answer to the Googlebot token. With no GoogleOther group, it follows your * group.
- Don't expect Google-Extended to cover it. Google-Extended is a control token for Gemini training and grounding with no crawler of its own; GoogleOther is a separate crawler on Google's list.
- Verify claimed hits: reverse DNS to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com, then forward-confirm, or match common-crawlers.json (317 prefixes, created November 13, 2026, when fetched November 14).
- Search your logs for the token, not just Googlebot: GoogleOther sends a mobile (Nexus 5X) or desktop Chrome string with "compatible; GoogleOther" in it.
- Keep dated copies of your robots.txt. Google doesn't say which team ran a given GoogleOther crawl or why, so your file is the record of what you allowed and when.
What does Google use the GoogleOther crawler for?
Google's list of common crawlers (developers.google.com, last updated October 30, 2026) calls GoogleOther "the generic crawler that may be used by various product teams for fetching publicly accessible content from sites", for example "one-off crawls for internal research and development", and says crawling preferences addressed to it "don't affect any specific product."
- GoogleOther-Image and GoogleOther-Video are versions of it tuned for image and video URLs, sending GoogleOther-Image/1.0 and GoogleOther-Video/1.0. Each lists two robots.txt tokens, its own and GoogleOther. GoogleOther itself sends a Chrome user agent, mobile (Nexus 5X) or desktop, containing "compatible; GoogleOther".
- For contrast, the same page says rules for Googlebot affect Google Search, including Discover and all Search features, plus products such as Google Images, Video mode and Google News. Those are the rules that decide Search; a GoogleOther group sits apart from them.
Which robots.txt group does GoogleOther follow?
Google's robots.txt specification (last updated October 23, 2026) says only one group is valid for a crawler: Google's crawlers use the group with the most specific user agent that matches and ignore the others. GoogleOther's only token is GoogleOther, so a Googlebot group never applies to it, and with no GoogleOther group it uses the * group.
- That's how it usually gets blocked or let in by accident. A file whose * group says Disallow: / while a Googlebot group opens the site keeps Search open and shuts GoogleOther out; an open * group with a restrictive Googlebot group does the reverse.
- Big sites that name it mostly shut it out. Fetched on November 14, 2026, amazon.com, cnn.com, ft.com, medium.com, nature.com, newyorker.com, usatoday.com and wired.com each had a group naming GoogleOther with Disallow: /, while reuters.com listed it in a shared group that disallows only specific paths.
How do you verify GoogleOther, and what does blocking it cost?
Google says its common crawlers always obey robots.txt rules when crawling automatically, generally crawl from the ranges in common-crawlers.json, and have reverse DNS names matching crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Its verification page gives the manual check: a reverse DNS lookup on the IP, a check that the name is in googlebot.com, google.com or googleusercontent.com, then a forward lookup that returns the same IP.
- On November 14, 2026 common-crawlers.json held 317 prefixes with a creationTime of November 13, 2026, so pull it on a schedule. A GoogleOther user agent from an address outside it that also doesn't survive the DNS round trip isn't Google's, and blocking it costs nothing.
- Blocking the real one costs whatever those product-team crawls would have used your pages for, and Google doesn't list those uses. If you're happy for Google teams to read your public pages, leave it open; if not, a GoogleOther group is the documented switch, and it leaves Googlebot alone.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Before we add a GoogleOther rule, what does our robots.txt tell the AI crawlers today, and which of them fall through to our * group?”
{
"tool": "read_ai_policy",
"arguments": {
"domain": "example.com"
}
}- Free, no key, cached about 6 hours. For 16 AI crawler, search and user-fetch tokens (Google-Extended, GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, CCBot and others) it reports whether the path is allowed and which group matched; matched_group "*" is the fall-through this page describes. It also reports Content-Signal lines, ai.txt, W3C TDMRep and whether llms.txt exists.
- It doesn't check GoogleOther or Googlebot yet, so read those groups yourself, and it reports what your files say, not what Google has crawled.
Rightshold reads a site's public crawl and AI-use signals. It isn't affiliated with Google, can't see what GoogleOther crawls are used for and can't make any company honor a preference. Crawler descriptions, user-agent strings and IP ranges are as Google published them, and the robots.txt files described were fetched, on November 14, 2026; they may change. Not legal advice.
What are the sources?
- Google: List of Google's common crawlers (GoogleOther)
- Google: How Google interprets the robots.txt specification
- Google: Verify requests from Google crawlers and fetchers
- Google: Common crawlers IP ranges (JSON)
Related
- What is Google-CloudVertexBot, and does blocking it affect Google Search?
- How to keep your site out of Google AI Overviews without leaving Google Search
- How to block AI training crawlers in robots.txt without leaving AI search
- Why an AI training opt-out or robots.txt is not access control
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'