# Google-CloudVertexBot is showing up in my server logs. What is it, who asked it to crawl my site, does blocking it hurt my Google Search rankings, and how do I tell it apart from Googlebot?

**Short answer:** Google-CloudVertexBot is the crawler behind website data stores in Google Cloud's Agent Search, which businesses use to build their own search and AI agents over web pages. Google's crawler list says rules addressed to it affect crawls requested by site owners for building Vertex AI Agents and have no effect on Google Search or other products, so disallowing it doesn't touch your rankings. It answers to two robots.txt tokens, Google-CloudVertexBot and Googlebot, and Google's crawlers use the most specific group that matches: with no Google-CloudVertexBot group, your Googlebot group applies to it. Google says it always obeys robots.txt on automatic crawls, crawls from the ranges in common-crawlers.json, and resolves to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com in reverse DNS, so you can verify it like Googlebot. If you run Agent Search on your own site, the opposite applies: Google says the bot must be able to reach your pages, including paywalled ones, while your sitemap is fetched by Googlebot, so a firewall that admits one but not the other quietly breaks indexing.

## Checklist
1. Read the token, not just the vendor: Google-CloudVertexBot crawls for Agent Search website data stores. Google says rules for it have no effect on Google Search or other products.
2. To refuse it, add "User-agent: Google-CloudVertexBot" followed by "Disallow: /". Google Search keeps using Googlebot and your Googlebot rules.
3. Watch the fallback: it also answers to the Googlebot token. With no Google-CloudVertexBot group, it follows your Googlebot group; with neither, it follows *.
4. Don't expect Google-Extended to cover it. Google-Extended governs Gemini training and grounding; Google-CloudVertexBot is a separate token on Google's list, and a group for one doesn't name the other.
5. Verify claimed hits the way you verify Googlebot: reverse DNS to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com, forward-confirm, or match common-crawlers.json (317 prefixes on November 11, 2026).
6. If you're building Agent Search over your own site, allow Google-CloudVertexBot to reach the pages you include, paywalled ones too, and allow Googlebot to fetch the sitemap. Check your CDN and firewall as well as robots.txt.
7. Exclude dynamic and duplicate URL patterns before you create the data store. Google bills each unique URL as a document and says it doesn't refund excess indexing charges from wrong patterns.
8. Keep dated copies of your robots.txt and any edge rules. If you later need to show what you told this crawler and when, that's the record.

## What is Google-CloudVertexBot, and who asks it to crawl your site?
- Google's list of common crawlers (developers.google.com, last updated October 30, 2026) gives Google-CloudVertexBot as the user-agent substring and lists two robots.txt tokens for it, Google-CloudVertexBot and Googlebot. It says crawling preferences addressed to it "affect crawls requested by the site owners' for building Vertex AI Agents" and that "it has no effect on Google Search or other products."
- Google Cloud's Agent Search docs (Prepare data for ingesting, last updated November 10, 2026) explain what those crawls feed. When someone creates a website data store, they give the URL patterns Google should crawl and index for search or recommendations, and Agent Search uses Google-CloudVertexBot to crawl and index the pages. The full user agent shown there is a mobile Chrome string ending in "(compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search)".
- Basic website search takes up to 50 included and 50 excluded URL patterns; advanced website indexing takes up to 500 of each, adds features such as search summarization and extractive answers, and requires verifying ownership of the domains you index. The docs say advanced indexing with automatic URL discovery indexes pages only if they're already known to Google.

## Do your Googlebot robots.txt rules apply to Google-CloudVertexBot, and why does Agent Search need Googlebot too?
- Google's robots.txt specification (last updated October 23, 2026) says only one group is valid for a crawler: Google's crawlers pick the group with the most specific user agent that matches, and other groups are ignored. Because Google-CloudVertexBot also carries the Googlebot token, a file with a Googlebot group and no Google-CloudVertexBot group hands it your Googlebot rules, which on most sites allow nearly everything.
- Publishers who want it out name it. Fetched on November 11, 2026, nytimes.com had a Google-CloudVertexBot group that disallowed the site except /wirecutter/ and /athletic/; bbc.co.uk and amazon.com each disallowed it entirely; theguardian.com included it in a shared group above Disallow: /; and theverge.com disallowed all but /sp/, with a second group for a shortened "CloudVertexBot" name that isn't on Google's list.
- If you use Agent Search on your own site, Google says to make sure Google-CloudVertexBot can access your content, including paywalled content, and to let your organization's upstream proxies pass the requests. It also says Agent Search fetches your sitemap with Googlebot, not Google-CloudVertexBot: if robots.txt, a CDN or a firewall admits Google-CloudVertexBot but blocks Googlebot, pages still get crawled but the sitemap can't be fetched and its URLs aren't indexed or refreshed through it.

## How do you verify Google-CloudVertexBot, and what do you lose by blocking it?
- Google says its common crawlers always obey robots.txt when crawling automatically, generally crawl from the ranges published in common-crawlers.json, and have hostnames matching crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. On November 11, 2026 the JSON file held 317 prefixes with a creation time the same day, so fetch it on a schedule rather than copying it once.
- A request whose user agent says Google-CloudVertexBot from an address that fails reverse and forward DNS, or sits outside that list, isn't Google's. Blocking a fake costs you nothing; blocking the real one costs you only inclusion in Agent Search data stores that list your URLs.
- That cost matters if you, or a partner you've agreed to, build an Agent Search app over your pages: pages disallowed to Google-CloudVertexBot can't be indexed into the data store. If that's a use you want, allow the paths it needs in a Google-CloudVertexBot group and keep everything else disallowed; the longest matching rule wins within the group.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "Before we add a Google-CloudVertexBot rule, what does our robots.txt tell Google-Extended and the other AI crawlers today, and which of them fall through to our * group?"

```json
{
  "tool": "read_ai_policy",
  "arguments": {
    "domain": "example.com"
  }
}
```
- Free, no key, cached about 6 hours. For 16 AI crawler, search and user-fetch tokens (Google-Extended, GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, CCBot and others) it reports whether the path is allowed and which group matched; matched_group "*" is the fall-through this page warns about. It also reports Content-Signal lines, ai.txt, W3C TDMRep and whether llms.txt exists.
- It doesn't check Google-CloudVertexBot or Googlebot yet, so read those groups yourself, and it reports what your files say, not what Google has crawled or indexed for anyone's data store.

_Rightshold reads a site's public crawl and AI-use signals. It isn't affiliated with Google, can't see which Agent Search data stores include your pages and can't make any company honor a preference. Crawler descriptions, user-agent strings, limits and IP ranges are as Google published them, and the robots.txt files described were fetched, on November 11, 2026; they may change. Not legal advice._

## Sources
- [Google: List of Google's common crawlers (Google-CloudVertexBot)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Google Cloud: Agent Search, Prepare data for ingesting (website data)](https://docs.cloud.google.com/generative-ai-app-builder/docs/prepare-data)
- [Google Cloud: Agent Search, Create a search data store (website content)](https://docs.cloud.google.com/generative-ai-app-builder/docs/create-data-store-es)
- [Google: How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
- [Google: Common crawlers IP ranges (JSON)](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)

## Related
- [How to keep your site out of Google AI Overviews without leaving Google Search](https://www.rightshold.com/answers/how-to-keep-your-site-out-of-google-ai-overviews-without-leaving-google-search)
- [How to block AI training crawlers in robots.txt without leaving AI search](https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search)
- [Does Cloudflare's AI bot blocking also block Googlebot and AI search?](https://www.rightshold.com/answers/does-cloudflares-ai-bot-blocking-also-block-googlebot-and-ai-search)
- [Why an AI training opt-out or robots.txt is not access control](https://www.rightshold.com/answers/why-an-ai-training-opt-out-or-robots-txt-is-not-access-control)

_Not legal advice. Updated 2026-11-11. JSON: https://www.rightshold.com/answers/what-is-google-cloudvertexbot-and-does-blocking-it-affect-google-search.json_