Rightshold™ Sign up

Answers · AI crawlers

How much do AI crawlers actually cost a website in bandwidth, and how much traffic do they send back?

Last updated

TL;DR: It depends on what you host, but the published cases are real money for small operators, and the traffic sent back is small. Read the Docs reported in July 2024 that one crawler downloaded 73 TB of zipped HTML in May 2024, almost 10 TB in a single day, costing over $5,000 in bandwidth charges; a crawler bug made it re-download the same unchanged files hundreds of times, and the company later covered the cost. After blocking the bots Cloudflare identifies as AI crawlers, Read the Docs' download bandwidth fell 75%, from about 800 GB to about 200 GB a day, which it put at around $1,500 a month if that traffic hit its origin. The Wikimedia Foundation reported in April 2025 that bandwidth for multimedia downloads had grown 50% since January 2024, mostly from scrapers, and that at least 65% of its most expensive traffic came from bots, although bots made up about 35% of pageviews. The reason is caching: bots bulk-read old and unpopular pages that aren't in the CDN cache, so they hit the origin. On the return side, Cloudflare's July 2025 data put Anthropic at about 38,000 pages crawled per referred visit, OpenAI at about 1,091, Perplexity at about 195 and Google at about 5.4. Measure your own bill before you decide what to block.

What should I check, step by step?

1. Pull a month of CDN and origin logs and sum bytes served by user agent, split into cache hits and requests that reached your origin. Origin bytes are what you pay for; cached HTML usually isn't the problem.

  1. Verify the big bot names before you blame them: match the claimed user agent to the operator's published IP list or reverse DNS. Scrapers borrow GPTBot and Googlebot names.
  2. Look for the expensive paths first: large downloads (zip, PDF, video, images), old URLs that redirect to uncached endpoints, search pages, git history and other dynamic pages. Read the Docs' 73 TB came through an old redirect to an uncached download URL.
  3. Send ETag and Last-Modified headers and answer conditional requests with 304, so a well-behaved crawler only re-downloads what changed. Read the Docs said the crawler that cost it $5,000 didn't use them.
  4. Rate limit per verified bot and per path at your edge, not just per IP. Read the Docs said IP rate limits didn't work because the crawlers came from many addresses.
  5. Use robots.txt for crawlers that honor it, and an edge rule for the rest. Blocking training crawlers doesn't remove you from Google Search; Googlebot and Google-Extended are separate controls.
  6. Compare what each crawler costs with what it sends back: count referred visits per AI platform in your analytics next to its crawl volume, and remember native apps often send no Referer, so referrals are undercounted.
  7. If others need your data in bulk, give them a cheaper channel such as a dump, feed or rate-limited API, and ask them to identify themselves to get higher limits. Wikimedia's rule of thumb is that stronger identification earns a higher limit.

How much have AI crawlers cost websites in bandwidth?

Read the Docs, July 2024: one crawler pulled 73 TB of zipped HTML in May 2024 and almost 10 TB in a single day, over $5,000 in bandwidth charges. Read the Docs said the crawler had no bandwidth limiting and didn't use ETags or Last-Modified to skip unchanged files; the company was responsive and covered the cost. In June 2024 another 10 TB, mostly zipped HTML and PDFs, came through Facebook's content downloader; Read the Docs' email to the contact in the bot's user agent bounced.

Why does AI crawler traffic cost more to serve than human visitors?

People cluster on the same popular pages, which stay in the CDN cache. Crawlers bulk-read everything, including old and rarely visited pages, so their requests miss the cache and go to your origin, where you pay for bandwidth and compute. Wikimedia and Read the Docs both describe this.

How many visitors do AI crawlers send back compared with how much they crawl?

Cloudflare's crawl-to-refer ratio counts HTML pages a platform's bots fetch per page visit it refers. For July 2025 Cloudflare reported Anthropic at about 38,000 to 1 (down from about 287,000 to 1 in January; Cloudflare suggests Claude's web search with cited links may be helping), OpenAI about 1,091 to 1, Perplexity about 195 to 1, Microsoft about 41 to 1 and Google about 5.4 to 1. Cloudflare also found training made up nearly 80% of AI bot crawling.

How do I ask Rightshold about this (free, no key)?

Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “What does my site's robots.txt currently tell GPTBot, ClaudeBot, CCBot, Google-Extended and the AI search and user-fetch bots, so I know which crawlers I've asked to stay away before I set edge limits?”

One-click setup

{
  "tool": "read_ai_policy",
  "arguments": {
    "domain": "example.com"
  }
}

Rightshold reads a site's public AI and crawl signals and checks agent signatures you send it. It doesn't see your traffic, calculate your hosting costs, rate limit or block crawlers. The figures above are from the cited reports for the dates given and won't match any other site's costs.

What are the sources?

Related

Not legal advice. JSON · Markdown · All answers

Get a free API key in one call

No card and no email needed. 100 free credits; the license and policy checks stay free without a key.

curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_...   # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
  -H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'
Get a keyQuickstart