If an AI crawler or agent is a Cloudflare Verified bot, does that mean it obeys my robots.txt and rate limits? What happens when a verified bot ignores robots.txt or keeps hammering my site after a 429, and can it lose its verified status?

What should I check, step by step?

1. Read Verified as "Cloudflare knows who this is and expects it to behave", not as "this request respects my robots.txt". Cloudflare judges sustained patterns, so a single bad crawl can get through.

2. Set the AI Crawler and Agent access policies (Search, Agent, Training) yourself instead of relying on defaults. Cloudflare's docs say Verified bots have historically been excluded in default bot configurations across all plans.
3. If you want your robots.txt to match those policies, turn on Bot Preference Sync. It prepends managed rules to your existing file and keeps your own Disallow lines, but it works by category and doesn't read your custom firewall rules, so per-company exceptions need the sync off and a hand-written file.
4. Enforce what matters with a block, not a preference: Cloudflare's own robots.txt docs say compliance is voluntary and point to AI Crawl Control for enforcement.
5. When a bot overloads you, answer with HTTP 429 and a Retry-After header. Cloudflare says whether a Verified bot slows down or pauses after a 429 is one of the signals behind its trust band.
6. Use crawl-delay only for crawlers that read it. Cloudflare counts an AI crawler ignoring crawl-delay as a policy breach, but Google's robots.txt documentation says Google doesn't support crawl-delay, so it won't slow Googlebot.
7. For agents labelled Intermediary (a service many end users operate), remember you're trusting the operator, not each user behind it. Cloudflare says it's experimenting with forwarding end-user information in the Forwarded header.
8. Keep your own logs of what each verified bot requested against what your robots.txt said at the time, with dated copies of the file. That's your evidence if you report a bot to Cloudflare or its operator.

What does a bot have to do to be a Cloudflare Verified bot?

Cloudflare's Verified bots page, last updated November 2, 2026, defines a Verified bot as one Cloudflare has confirmed is transparent about who it is and what it does. It must declare who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and it must behave: obey robots.txt and crawl directives, keep reasonable request rates, and not have been observed evading website owner preferences or attacking sites.

• Since July 1, 2026, signed agents count as Verified too. Cloudflare labels each one Direct (one narrow operator, usually on its own infrastructure) or Intermediary (an agentic service many end users can operate), and classifies behavior as Search, Agent, Training, Transact and other categories. Search, Agent and Training are the managed presets site owners can act on in every plan.
• The October 16, 2026 blog post restates three criteria for Verified status: identify yourself accurately and securely, do only what you say you will do, and don't engage in abusive behavior. It notes that a robots.txt file sets the minimum rules, and that what isn't forbidden there isn't automatically allowed if a site signals in other ways that access is unwelcome.

Can a Cloudflare Verified bot lose its status for ignoring robots.txt or 429s?

The policy says a service that breaches any validation requirement is removed from the global allowlist. Its examples include IPs not used solely by the verified service, unpatched vulnerabilities, a disclosed purpose that doesn't match the traffic, and an AI crawler that doesn't respect crawl-delay in robots.txt. Cloudflare may revoke Verified status for all of an organization's bots and agents after repeated violations, including complying with the policy for one bot while violating it with another.

• There's precedent. In August 2025 Cloudflare reported that Perplexity's declared crawler was followed by undeclared, browser-like requests when sites blocked it, and said it removed Perplexity from its verified bots. The October 2026 post cites that de-listing as an example.
• The new piece is continuous measurement. Cloudflare says it now evaluates how often each Verified bot respects robots.txt and whether its traffic slows or pauses after a 429, looking for repeated patterns rather than isolated events, and turns that into an explainable trust band rather than a permanent judgment. As of the post, it was using the system internally to build history; publishing scores on Radar and letting customers choose the trust threshold that decides how Verified bot traffic is handled were described as plans.

Does Cloudflare's Verified bot label enforce my robots.txt?

It doesn't enforce your robots.txt on a given request. Cloudflare's robots.txt documentation says compliance is voluntary and that the file doesn't stop a crawler technically; enforcement is a separate control (AI Crawl Control, WAF rules or Bot Management).

• It doesn't decide what you want. Bot Preference Sync, announced August 21, 2026 (Cloudflare said it would be on by default for new customers), writes robots.txt rules from your Search, Agent and Training choices so the file and the edge say the same thing, but only by category. With Training set to Disallow, Cloudflare says cooperating mixed-use crawlers that meet its transparency step can still reach your content for search indexing.
• And it doesn't settle disputes. A trust band is Cloudflare's assessment of a pattern across many sites. If a bot ignored your rules, what you can show is your own logs of the requests set against the robots.txt you served at that time.