Answers · AI crawlers
If I turn on Cloudflare's AI crawler blocking, will it also block Googlebot, Bing or AI search?
Last updated
TL;DR: It depends which setting you pick, and the meaning changed on September 15, 2026. Cloudflare's AI Crawler and Agent access policies (Security > Settings) have three separate rows: Search, Agent and Training. On the Training row, Block and Block on pages with ads now also apply to mixed-use crawlers, which Cloudflare says include Googlebot, Bingbot and Applebot, so they cut you out of those search engines too (everywhere, or on pages with ads). To stop training and stay in search, choose Disallow AI Training: Cloudflare writes a no-training preference into your robots.txt, keeps letting Apple's, Google's and Microsoft's mixed-use crawlers in for search, and blocks other training crawlers at its edge, including the training-only crawlers from Amazon, Anthropic, Meta and OpenAI. AI search crawlers and assistant fetches follow the Search and Agent rows, not Training. One surprise for older accounts: sites that had the old Block AI Bots switch on were migrated to Search Allow, Training Disallow AI Training and Agent Block on pages with ads.
What should I check, step by step?
1. Open Security > Settings, filter by Bot traffic, select AI Crawler and Agent access policies and read all three rows, not just Training.
- If Training says Block or Block on pages with ads and you want search traffic, switch it to Disallow. Cloudflare says Block stops Applebot, Bingbot and Googlebot from reaching your site, search included.
- Leave Search on Allow unless you really want out of search. Cloudflare says fewer than 1% of its sites block Search bots.
- Check the Agent row. Agent covers automated activity for a person in real time, such as chat fetch bots and browser-use agents. Block on pages with ads stops those on pages where Cloudflare detects ads. Personal Agents inherit the Agent policy unless you set them one by one, and Cloudflare recommends Allow for them.
- Turn on Bot Preference Sync so robots.txt says what the edge enforces. It prepends a managed block and the Content Signals Policy to your existing file and keeps your own rules. Reload /robots.txt to check; Cloudflare says it appears within seconds with no cache purge.
- Expect Search Console to flag Content-signal lines as 'Syntax not understood' now and then. Cloudflare says it has seen no effect on crawl rate or SEO from that.
- Bing: until Microsoft supports a robots.txt no-training preference (targeted for early 2027, per Cloudflare), Disallow AI Training doesn't tell Bing anything. Cloudflare points to Bing's NOARCHIVE meta tag, or its Block URLs or Content Removal tool, for now.
- Have custom exceptions, like a crawler you've licensed? Bot Preference Sync doesn't read individual custom rules. Turn sync off and keep your own robots.txt in line with your WAF rules.
- Want out of Google's AI Overviews but not Search? None of these settings does that. Use Google's Search generative AI control in Search Console.
What do Cloudflare's Allow, Disallow AI Training and Block settings do for training crawlers?
Allow: all crawlers allowed unless another setting or a WAF rule blocks them.
- Disallow AI Training: publishes a no-training preference in robots.txt. Mixed-use crawlers that Cloudflare calls Accountable (Applebot, Bingbot and Googlebot) stay allowed for search; every other training crawler is blocked at the edge. Cloudflare says blocking the training-only crawlers of Amazon, Anthropic, Meta and OpenAI doesn't affect their search crawlers. Mixed-use crawlers are excluded from this edge block.
- Block on pages with ads: training crawlers, mixed-use ones included, are blocked on pages where Cloudflare detects ads.
- Block: training crawlers, mixed-use ones included, are blocked everywhere. That means Googlebot, Bingbot and Applebot can't reach the site for search either.
How were old Cloudflare Block AI Bots settings migrated?
Old Block AI Bots set to Block or Block on pages with ads became: Search Allow, Training Disallow AI Training, Agent Block on pages with ads. Left off, it became Allow on all three.
- If you had already used the Search/Training/Agent rows, a Training Block or Block on pages with ads became Disallow AI Training; Search and Agent kept their values. So an old Training Block didn't silently start blocking Googlebot.
- New domains since September 15 get a preset at onboarding. If the site makes money from ads: Training Disallow AI Training and Agent Block on pages with ads. If not: all Allow. Bot Preference Sync is on in both, and you can change any of it later.
What does Cloudflare's Bot Preference Sync add to my robots.txt?
With sync on and at least one row set to Block, Block on pages with ads or (Training) Disallow, Cloudflare prepends a managed block. Its documented example has a User-Agent: * group with 'Content-signal: search=yes, ai-train=no, use=reference' and Allow: /, then Disallow: / groups for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. Cloudflare updates that bot list from its own bot tracking over time.
- Your own User-agent: * rules still count. RFC 9309 says when more than one group matches a crawler, their rules are combined into one, so the managed * group's Allow: / sits alongside your Disallow lines and the most specific match still decides each path.
- robots.txt is a request. Cloudflare's docs say crawlers can ignore it, which is why the edge block exists, and why a crawler shown as allowed in robots.txt may still be blocked by a separate rule.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “After changing my Cloudflare AI crawler settings, which AI crawlers does my live robots.txt now disallow, and what Content-Signal does it publish?”
{
"tool": "read_ai_policy",
"arguments": {
"domain": "example.com",
"path": "/"
}
}- Free, no key, cached 6 hours, so check again later if you've only just saved the change. For each AI crawler and user-fetch agent it says whether the path is allowed, which robots.txt group matched and the Content-Signal values (search, ai-input, ai-train) that apply.
- It reads the published robots.txt only. It can't see Cloudflare's edge blocks, WAF rules or dashboard settings.
Rightshold isn't affiliated with Cloudflare. It reads your published robots.txt and AI signals on request and doesn't see or change your Cloudflare settings or traffic. Setting names and behavior above are from Cloudflare's documentation and blog as of November 9, 2026 and can change.
What are the sources?
- Cloudflare Blog: Have it both ways: stay discoverable in search while disallowing AI training (September 15, 2026; Block now applies to Googlebot, Bingbot, Applebot; migration tables)
- Cloudflare Blog: Say it once: introducing Bot Preference Sync (August 21, 2026)
- Cloudflare docs: AI Crawler and Agent access policies (Search, Agent, Training; actions; Personal Agents)
- Cloudflare docs: robots.txt setting (Bot Preference Sync, managed directives, Content Signals, Search Console warning)
- RFC 9309: Robots Exclusion Protocol (matching groups are combined)
- Google Search Console Help: Search generative AI control
Related
- Keeping your site out of Google AI Overviews without leaving Google Search
- How to block AI training crawlers in robots.txt without leaving AI search
- How to charge AI crawlers per request with HTTP 402 without blocking search crawlers
- How to block spoofed ChatGPT-User and Perplexity-User requests at the edge without blocking the real ones
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'