# If an AI crawler or agent is a Cloudflare Verified bot, does that mean it obeys my robots.txt and rate limits? What happens when a verified bot ignores robots.txt or keeps hammering my site after a 429, and can it lose its verified status?

**Short answer:** It's supposed to, but the label isn't a guarantee for your site on any given request. Cloudflare's Verified bots page (updated November 2, 2026) sets two bars: honest self-identification (a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS) and non-abusive behavior, meaning it obeys robots.txt and crawl directives, keeps reasonable request rates and hasn't been seen evading site owners' preferences. Breaching the policy gets a service removed from the global allowlist; Cloudflare's examples include an AI crawler that doesn't respect the crawl-delay directive, and it may revoke Verified status for every bot an organization runs after repeated violations. Cloudflare de-listed Perplexity's bot in 2025 after it tried to reach sites that had opted out and evaded blocks. On October 16, 2026 Cloudflare said it now measures, internally for now, whether each Verified bot follows robots.txt and slows down after an HTTP 429, and turns repeated patterns into a trust band; one-off violations may not count. It plans to publish the scores on Radar and let customers pick a trust threshold, but that wasn't available yet. Because Verified bots have historically been excluded from default bot protections, set your AI crawler policies explicitly, send 429 with Retry-After when you need a bot to slow down, and enforce blocks at the edge rather than trusting the label.

## Checklist
1. Read Verified as "Cloudflare knows who this is and expects it to behave", not as "this request respects my robots.txt". Cloudflare judges sustained patterns, so a single bad crawl can get through.
2. Set the AI Crawler and Agent access policies (Search, Agent, Training) yourself instead of relying on defaults. Cloudflare's docs say Verified bots have historically been excluded in default bot configurations across all plans.
3. If you want your robots.txt to match those policies, turn on Bot Preference Sync. It prepends managed rules to your existing file and keeps your own Disallow lines, but it works by category and doesn't read your custom firewall rules, so per-company exceptions need the sync off and a hand-written file.
4. Enforce what matters with a block, not a preference: Cloudflare's own robots.txt docs say compliance is voluntary and point to AI Crawl Control for enforcement.
5. When a bot overloads you, answer with HTTP 429 and a Retry-After header. Cloudflare says whether a Verified bot slows down or pauses after a 429 is one of the signals behind its trust band.
6. Use crawl-delay only for crawlers that read it. Cloudflare counts an AI crawler ignoring crawl-delay as a policy breach, but Google's robots.txt documentation says Google doesn't support crawl-delay, so it won't slow Googlebot.
7. For agents labelled Intermediary (a service many end users operate), remember you're trusting the operator, not each user behind it. Cloudflare says it's experimenting with forwarding end-user information in the Forwarded header.
8. Keep your own logs of what each verified bot requested against what your robots.txt said at the time, with dated copies of the file. That's your evidence if you report a bot to Cloudflare or its operator.

## What does a bot have to do to be a Cloudflare Verified bot?
- Cloudflare's Verified bots page, last updated November 2, 2026, defines a Verified bot as one Cloudflare has confirmed is transparent about who it is and what it does. It must declare who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and it must behave: obey robots.txt and crawl directives, keep reasonable request rates, and not have been observed evading website owner preferences or attacking sites.
- Since July 1, 2026, signed agents count as Verified too. Cloudflare labels each one Direct (one narrow operator, usually on its own infrastructure) or Intermediary (an agentic service many end users can operate), and classifies behavior as Search, Agent, Training, Transact and other categories. Search, Agent and Training are the managed presets site owners can act on in every plan.
- The October 16, 2026 blog post restates three criteria for Verified status: identify yourself accurately and securely, do only what you say you will do, and don't engage in abusive behavior. It notes that a robots.txt file sets the minimum rules, and that what isn't forbidden there isn't automatically allowed if a site signals in other ways that access is unwelcome.

## Can a Cloudflare Verified bot lose its status for ignoring robots.txt or 429s?
- The policy says a service that breaches any validation requirement is removed from the global allowlist. Its examples include IPs not used solely by the verified service, unpatched vulnerabilities, a disclosed purpose that doesn't match the traffic, and an AI crawler that doesn't respect crawl-delay in robots.txt. Cloudflare may revoke Verified status for all of an organization's bots and agents after repeated violations, including complying with the policy for one bot while violating it with another.
- There's precedent. In August 2025 Cloudflare reported that Perplexity's declared crawler was followed by undeclared, browser-like requests when sites blocked it, and said it removed Perplexity from its verified bots. The October 2026 post cites that de-listing as an example.
- The new piece is continuous measurement. Cloudflare says it now evaluates how often each Verified bot respects robots.txt and whether its traffic slows or pauses after a 429, looking for repeated patterns rather than isolated events, and turns that into an explainable trust band rather than a permanent judgment. As of the post, it was using the system internally to build history; publishing scores on Radar and letting customers choose the trust threshold that decides how Verified bot traffic is handled were described as plans.

## Does Cloudflare's Verified bot label enforce my robots.txt?
- It doesn't enforce your robots.txt on a given request. Cloudflare's robots.txt documentation says compliance is voluntary and that the file doesn't stop a crawler technically; enforcement is a separate control (AI Crawl Control, WAF rules or Bot Management).
- It doesn't decide what you want. Bot Preference Sync, announced August 21, 2026 (Cloudflare said it would be on by default for new customers), writes robots.txt rules from your Search, Agent and Training choices so the file and the edge say the same thing, but only by category. With Training set to Disallow, Cloudflare says cooperating mixed-use crawlers that meet its transparency step can still reach your content for search indexing.
- And it doesn't settle disputes. A trust band is Cloudflare's assessment of a pattern across many sites. If a bot ignored your rules, what you can show is your own logs of the requests set against the robots.txt you served at that time.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "This request says it's a verified AI agent and carries Web Bot Auth Signature headers. Does the signature verify, and is the signer one we've chosen to let past our crawl rules?"

```json
{
  "tool": "gate_check",
  "arguments": {
    "policy": {
      "require_web_bot_auth": true,
      "allowed_signature_agents": [
        "chatgpt.com"
      ]
    },
    "request": {
      "method": "GET",
      "url": "<the requested URL>",
      "headers": {
        "user-agent": "<as received>",
        "signature-agent": "<as received>",
        "signature-input": "<as received>",
        "signature": "<as received>"
      }
    }
  }
}
```
- Free, no key. It resolves the signer's HTTPS key directory and returns allow or deny with reasons, plus the signer it resolved. allowed_signature_agents narrows it to signers you name; drop it to accept any signer whose signature verifies.
- It checks the signature on this request against your policy. It doesn't read Cloudflare's Verified list or trust bands, doesn't check IP lists or reverse DNS, and can't tell whether a bot has been following your robots.txt over time; your logs do that.

_Rightshold isn't affiliated with Cloudflare or any crawler operator named here, can't see Cloudflare's Verified bot decisions or trust bands and can't make a bot honor robots.txt. Policies and plans are as Cloudflare described them on its Verified bots page (updated November 2, 2026) and in its October 16, 2026 and August 21, 2026 blog posts, checked on November 11, 2026; Cloudflare may change them. Not legal advice._

## Sources
- [Cloudflare Docs: Verified bots (policy, classification, breach of policy; updated November 2, 2026)](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/policy/)
- [Cloudflare Blog: Continuous accountability for Verified bots (October 16, 2026)](https://blog.cloudflare.com/verified-bots-accountability/)
- [Cloudflare Blog: Say it once: introducing Bot Preference Sync (August 21, 2026)](https://blog.cloudflare.com/bot-preference-sync/)
- [Cloudflare Docs: robots.txt setting (Bot Preference Sync, Content Signals Policy)](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)
- [Cloudflare Blog: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (August 4, 2025)](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/)
- [Google Search Central: How Google interprets the robots.txt specification (crawl-delay not supported)](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
- [IETF: RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)

## Related
- [Does Cloudflare's AI bot blocking also block Googlebot and AI search?](https://www.rightshold.com/answers/does-cloudflares-ai-bot-blocking-also-block-googlebot-and-ai-search)
- [How to verify which AI agent is hitting your site with Web Bot Auth](https://www.rightshold.com/answers/how-to-verify-which-ai-agent-is-hitting-your-site-with-web-bot-auth)
- [How to slow down AI crawlers overloading your server without hurting Google rankings](https://www.rightshold.com/answers/how-to-slow-down-ai-crawlers-overloading-your-server-without-hurting-google-rankings)
- [How to catch crawlers that ignore robots.txt with a disallowed trap URL](https://www.rightshold.com/answers/how-to-catch-crawlers-that-ignore-robots-txt-with-a-disallowed-trap-url)

_Not legal advice. Updated 2026-11-11. JSON: https://www.rightshold.com/answers/does-a-cloudflare-verified-bot-have-to-obey-your-robots-txt-and-can-it-lose-verified-status.json_