{
  "question": "How do I charge AI crawlers per request with HTTP 402 without blocking Google and other search crawlers?",
  "slug": "how-to-charge-ai-crawlers-per-request-with-http-402-without-blocking-search-crawlers",
  "url": "https://www.rightshold.com/answers/how-to-charge-ai-crawlers-per-request-with-http-402-without-blocking-search-crawlers",
  "short_answer": "Decide per crawler, verify before you decide, and keep search crawlers on plain 200s. Google says every 4xx except 429 tells its crawlers the content doesn't exist: new URLs aren't indexed and indexed ones are removed, so a 402 sent to Googlebot works as a deindex, not an invoice. Verify each claimed bot first (its operator's published IP list, a reverse-then-forward DNS check, or a Web Bot Auth signature), then give it Allow, Charge or Block. Allow Googlebot, Bingbot and the AI search crawlers you want to be cited by; charge the training and bulk AI crawlers that can pay; keep people on the normal page; and leave robots.txt, sitemap.xml and your discovery pages free. Make the 402 a price a program can read: Cloudflare's Pay Per Crawl (closed beta) sends a crawler-price header, and x402 sends a base64 PAYMENT-REQUIRED header. One limit to plan around: Google-Extended has no user agent of its own, so you can't charge Gemini separately from Google Search; it's a robots.txt preference only.",
  "checklist": [
    "Sort the crawlers that reach you into classes before writing any rule: search engines (Googlebot, Bingbot), AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot), fetches a person asked an assistant to make (ChatGPT-User, Claude-User, Perplexity-User), training crawlers (GPTBot, ClaudeBot, CCBot) and unknown automation. Pick Allow, Charge or Block for each class.",
    "Never send 402, 401 or 403 to a search crawler you want in the index. Google treats all 4xx except 429 as content that doesn't exist and removes indexed URLs that start returning them, and Cloudflare's Pay Per Crawl docs warn that setting search engine crawlers to Charge or Block may hurt your SEO.",
    "Verify before you charge or exempt. Match each user-agent token against its operator's own list (openai.com/gptbot.json, searchbot.json and chatgpt-user.json; perplexity.ai's perplexitybot.json and perplexity-user.json), use reverse-then-forward DNS for Googlebot (googlebot.com or google.com) and Bingbot (search.msn.com), or verify a Web Bot Auth signature. A request that only says it's Googlebot gets no exemption.",
    "Keep discovery free: robots.txt, sitemap.xml, the homepage and category pages. Cloudflare's Pay Per Crawl always leaves /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json free, and lets you exclude more URI patterns with Configuration Rules.",
    "Make the 402 machine-readable: put the amount and the way to pay in headers (crawler-price: USD 0.01 for Pay Per Crawl, PAYMENT-REQUIRED for x402) and say the same in a short body. RFC 9110 still lists 402 as reserved for future use, so the payment protocol, not HTTP, defines what the response means.",
    "Tie each payment to a verified identity. Pay Per Crawl requires the crawler to sign with Web Bot Auth and to include crawler-exact-price or crawler-max-price among the signed components, so a request with a borrowed name can't pay, or run up a bill, as someone else.",
    "Keep humans out of the 402 path. Browsers keep getting the page they always got. If you also paywall people, mark the locked part with isAccessibleForFree: false and hasPart.cssSelector, which Google says helps it tell a paywall from cloaking.",
    "Log every decision (claimed crawler, how it was verified, action, price, status) and watch Search Console's indexing and crawl reports for a week after launch. More 4xx responses to Googlebot means a search crawler fell into the charge path."
  ],
  "sections": [
    {
      "heading": "Why a 402 to a search crawler deindexes the page",
      "points": [
        "Google's documentation on HTTP status codes says Google doesn't use the content of URLs that return 4xx, that all 4xx codes except 429 are treated the same way (the content doesn't exist), and that URLs already in the index are removed. 429 is treated as server overload, which slows crawling. No status code charges Googlebot without either dropping the page or throttling the crawl.",
        "Google-Extended is the token Google offers for Gemini training and grounding, and Google says it has no separate HTTP user agent: crawling uses the existing Google user agents and the token works only in robots.txt. It doesn't affect inclusion in Google Search. So a 402 aimed at Gemini would hit Googlebot; the only lever for Gemini use is the robots.txt token.",
        "Google's paywalled-content guide says AI Overviews and AI Mode draw on web sources and are subject to Search's preview controls, so the levers for those are snippet and preview settings, not a payment status."
      ]
    },
    {
      "heading": "Allow, charge or block, crawler by crawler",
      "points": [
        "Cloudflare's Pay Per Crawl gives each crawler an action (Charge, Allow or Block) and one price for every crawler set to Charge; its FAQ says you can't set different prices for different crawlers. Dynamic pricing lets your origin or a Worker send a crawler-price header per path, and every origin request carries cf-pay-per-crawl, with pricing=bypass when the request isn't subject to payment. A WAF or Bot Management block overrides Charge. The docs list it as a closed beta, with Cloudflare acting as merchant of record.",
        "AI search crawlers are a business choice. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, and Perplexity says PerplexityBot surfaces sites in its search results and isn't used to crawl content for AI foundation models. A crawler that won't pay won't cite you either.",
        "User-triggered fetches bill one person's question at a time. OpenAI says ChatGPT-User visits a page when a user asks and isn't used to decide whether content appears in search. Charge them if the content is worth it per read, but expect a declined 402 to end in an answer built from someone else's page."
      ]
    },
    {
      "heading": "What the paying side needs to see",
      "points": [
        "Pay Per Crawl: the first response is 402 with crawler-price. The crawler retries with crawler-exact-price (matching the quoted price) or crawler-max-price, signed with Web Bot Auth. A success is 200 with crawler-charged; a failure is 402 with crawler-error such as InvalidCrawlerExactPrice. Error responses aren't billed, and every re-crawl of the same page is billed again, so crawlers need their own spend tracking.",
        "x402: the first response is 402 with a base64 PAYMENT-REQUIRED header naming the amount, asset, network and payTo address. The client retries with PAYMENT-SIGNATURE, and the server answers with the content plus PAYMENT-RESPONSE. A payment proves that someone paid, not who they are, so check identity separately if exemptions or prices depend on it."
      ]
    }
  ],
  "checks": [],
  "ask_rightshold": {
    "start": "https://www.rightshold.com/start?ref=seo",
    "mcp_endpoint": "https://www.rightshold.com/mcp",
    "ask": "A crawler hit my paid section and says it's a registered AI crawler. Does its Web Bot Auth signature verify, and which signer is it, before I send it a 402 price or let it through free?",
    "mcp_call": {
      "tool": "gate_check",
      "arguments": {
        "policy": {
          "require_web_bot_auth": true
        },
        "request": {
          "method": "GET",
          "url": "<the requested URL>",
          "headers": {
            "signature-agent": "<as received>",
            "signature-input": "<as received>",
            "signature": "<as received>",
            "crawler-max-price": "<as received, if sent>"
          }
        }
      }
    },
    "also": [
      "Free, no key. It resolves the signer's HTTPS key directory and returns allow or deny with reasons, plus the signer it resolved. Key your Allow, Charge or Block table on that signer, never on the user-agent. Allow answers can be cached for at most 5 minutes, and for less when the signature expires sooner.",
      "allowed_signature_agents narrows the policy to the signers you have priced, as host names or the exact https URL of one signer's JWKS or Signature Agent Card.",
      "gate_check doesn't check IP addresses or DNS. Keep the published-list or reverse DNS check for unsigned search crawlers such as Googlebot and Bingbot, and exempt them before any charge rule runs.",
      "The Gate page (/gate) has free CC0 middleware snippets that run the same check on incoming requests."
    ],
    "scope": "Rightshold checks the signatures and gate policies you send it. It doesn't sit in front of your site, set prices, collect payments from crawlers, or run Pay Per Crawl or x402 for you, and it doesn't verify IP addresses. Not legal advice."
  },
  "sources": [
    {
      "title": "Google: How HTTP status codes affect Google's crawlers",
      "url": "https://developers.google.com/search/docs/crawling-indexing/http-network-errors"
    },
    {
      "title": "Google: Common crawlers (Google-Extended has no separate user agent)",
      "url": "https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers"
    },
    {
      "title": "Google: Structured data for subscription and paywalled content",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/paywalled-content"
    },
    {
      "title": "Google: Verify requests from Google crawlers and fetchers",
      "url": "https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot"
    },
    {
      "title": "Cloudflare docs: What is Pay Per Crawl? (closed beta)",
      "url": "https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/"
    },
    {
      "title": "Cloudflare docs: Pay Per Crawl, crawl pages (crawler-price, crawler-exact-price, crawler-charged)",
      "url": "https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/"
    },
    {
      "title": "Cloudflare docs: Pay Per Crawl, select crawlers to charge (SEO warning)",
      "url": "https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge/"
    },
    {
      "title": "Cloudflare docs: Pay Per Crawl advanced configuration (free paths, dynamic pricing)",
      "url": "https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/advanced-configuration/"
    },
    {
      "title": "Cloudflare docs: Pay Per Crawl FAQ",
      "url": "https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/faq/"
    },
    {
      "title": "RFC 9110 section 15.5.3: 402 Payment Required",
      "url": "https://www.rfc-editor.org/rfc/rfc9110#section-15.5.3"
    },
    {
      "title": "x402 HTTP transport (v2): PAYMENT-REQUIRED, PAYMENT-SIGNATURE, PAYMENT-RESPONSE",
      "url": "https://github.com/x402-foundation/x402/blob/main/specs/transports-v2/http.md"
    },
    {
      "title": "OpenAI: Overview of OpenAI crawlers",
      "url": "https://platform.openai.com/docs/bots"
    },
    {
      "title": "Perplexity: Perplexity crawlers",
      "url": "https://docs.perplexity.ai/guides/bots"
    },
    {
      "title": "The Gate on Rightshold",
      "url": "https://www.rightshold.com/gate"
    }
  ],
  "related": [
    {
      "title": "How to accept x402 payments on an API",
      "url": "https://www.rightshold.com/answers/how-to-accept-x402-payments-on-an-api"
    },
    {
      "title": "How to read an x402 Payment Required response as a price",
      "url": "https://www.rightshold.com/answers/how-to-read-an-x402-payment-required-response-as-a-price"
    },
    {
      "title": "How to block AI training crawlers in robots.txt without leaving AI search",
      "url": "https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search"
    },
    {
      "title": "How to verify which AI agent is hitting your site with Web Bot Auth",
      "url": "https://www.rightshold.com/answers/how-to-verify-which-ai-agent-is-hitting-your-site-with-web-bot-auth"
    }
  ],
  "computed_from": "Curated guidance; license verdicts computed from Rightshold's license profiles at render time. Not legal advice.",
  "updated": "2026-11-07"
}