How do I charge AI crawlers per request with HTTP 402 without blocking Google and other search crawlers?

What should I check, step by step?

1. Sort the crawlers that reach you into classes before writing any rule: search engines (Googlebot, Bingbot), AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot), fetches a person asked an assistant to make (ChatGPT-User, Claude-User, Perplexity-User), training crawlers (GPTBot, ClaudeBot, CCBot) and unknown automation. Pick Allow, Charge or Block for each class.

2. Never send 402, 401 or 403 to a search crawler you want in the index. Google treats all 4xx except 429 as content that doesn't exist and removes indexed URLs that start returning them, and Cloudflare's Pay Per Crawl docs warn that setting search engine crawlers to Charge or Block may hurt your SEO.
3. Verify before you charge or exempt. Match each user-agent token against its operator's own list (openai.com/gptbot.json, searchbot.json and chatgpt-user.json; perplexity.ai's perplexitybot.json and perplexity-user.json), use reverse-then-forward DNS for Googlebot (googlebot.com or google.com) and Bingbot (search.msn.com), or verify a Web Bot Auth signature. A request that only says it's Googlebot gets no exemption.
4. Keep discovery free: robots.txt, sitemap.xml, the homepage and category pages. Cloudflare's Pay Per Crawl always leaves /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json free, and lets you exclude more URI patterns with Configuration Rules.
5. Make the 402 machine-readable: put the amount and the way to pay in headers (crawler-price: USD 0.01 for Pay Per Crawl, PAYMENT-REQUIRED for x402) and say the same in a short body. RFC 9110 still lists 402 as reserved for future use, so the payment protocol, not HTTP, defines what the response means.
6. Tie each payment to a verified identity. Pay Per Crawl requires the crawler to sign with Web Bot Auth and to include crawler-exact-price or crawler-max-price among the signed components, so a request with a borrowed name can't pay, or run up a bill, as someone else.
7. Keep humans out of the 402 path. Browsers keep getting the page they always got. If you also paywall people, mark the locked part with isAccessibleForFree: false and hasPart.cssSelector, which Google says helps it tell a paywall from cloaking.
8. Log every decision (claimed crawler, how it was verified, action, price, status) and watch Search Console's indexing and crawl reports for a week after launch. More 4xx responses to Googlebot means a search crawler fell into the charge path.

Why does returning HTTP 402 to Googlebot remove pages from Google Search?

Google's documentation on HTTP status codes says Google doesn't use the content of URLs that return 4xx, that all 4xx codes except 429 are treated the same way (the content doesn't exist), and that URLs already in the index are removed. 429 is treated as server overload, which slows crawling. No status code charges Googlebot without either dropping the page or throttling the crawl.

• Google-Extended is the token Google offers for Gemini training and grounding, and Google says it has no separate HTTP user agent: crawling uses the existing Google user agents and the token works only in robots.txt. It doesn't affect inclusion in Google Search. So a 402 aimed at Gemini would hit Googlebot; the only lever for Gemini use is the robots.txt token.
• Google's paywalled-content guide says AI Overviews and AI Mode draw on web sources and are subject to Search's preview controls, so the levers for those are snippet and preview settings, not a payment status.

Which crawlers should I charge with HTTP 402, and which should I let through free?

Cloudflare's Pay Per Crawl gives each crawler an action (Charge, Allow or Block) and one price for every crawler set to Charge; its FAQ says you can't set different prices for different crawlers. Dynamic pricing lets your origin or a Worker send a crawler-price header per path, and every origin request carries cf-pay-per-crawl, with pricing=bypass when the request isn't subject to payment. A WAF or Bot Management block overrides Charge. The docs list it as a closed beta, with Cloudflare acting as merchant of record.

• AI search crawlers are a business choice. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, and Perplexity says PerplexityBot surfaces sites in its search results and isn't used to crawl content for AI foundation models. A crawler that won't pay won't cite you either.
• User-triggered fetches bill one person's question at a time. OpenAI says ChatGPT-User visits a page when a user asks and isn't used to decide whether content appears in search. Charge them if the content is worth it per read, but expect a declined 402 to end in an answer built from someone else's page.

What headers does a paying AI crawler need in a 402 response, with Pay Per Crawl or x402?

Pay Per Crawl: the first response is 402 with crawler-price. The crawler retries with crawler-exact-price (matching the quoted price) or crawler-max-price, signed with Web Bot Auth. A success is 200 with crawler-charged; a failure is 402 with crawler-error such as InvalidCrawlerExactPrice. Error responses aren't billed, and every re-crawl of the same page is billed again, so crawlers need their own spend tracking.

• x402: the first response is 402 with a base64 PAYMENT-REQUIRED header naming the amount, asset, network and payTo address. The client retries with PAYMENT-SIGNATURE, and the server answers with the content plus PAYMENT-RESPONSE. A payment proves that someone paid, not who they are, so check identity separately if exemptions or prices depend on it.