{
  "question": "How much do AI crawlers actually cost a website in bandwidth, and how much traffic do they send back?",
  "slug": "how-much-do-ai-crawlers-cost-a-website-in-bandwidth-and-what-do-they-send-back",
  "url": "https://www.rightshold.com/answers/how-much-do-ai-crawlers-cost-a-website-in-bandwidth-and-what-do-they-send-back",
  "short_answer": "It depends on what you host, but the published cases are real money for small operators, and the traffic sent back is small. Read the Docs reported in July 2024 that one crawler downloaded 73 TB of zipped HTML in May 2024, almost 10 TB in a single day, costing over $5,000 in bandwidth charges; a crawler bug made it re-download the same unchanged files hundreds of times, and the company later covered the cost. After blocking the bots Cloudflare identifies as AI crawlers, Read the Docs' download bandwidth fell 75%, from about 800 GB to about 200 GB a day, which it put at around $1,500 a month if that traffic hit its origin. The Wikimedia Foundation reported in April 2025 that bandwidth for multimedia downloads had grown 50% since January 2024, mostly from scrapers, and that at least 65% of its most expensive traffic came from bots, although bots made up about 35% of pageviews. The reason is caching: bots bulk-read old and unpopular pages that aren't in the CDN cache, so they hit the origin. On the return side, Cloudflare's July 2025 data put Anthropic at about 38,000 pages crawled per referred visit, OpenAI at about 1,091, Perplexity at about 195 and Google at about 5.4. Measure your own bill before you decide what to block.",
  "checklist": [
    "Pull a month of CDN and origin logs and sum bytes served by user agent, split into cache hits and requests that reached your origin. Origin bytes are what you pay for; cached HTML usually isn't the problem.",
    "Verify the big bot names before you blame them: match the claimed user agent to the operator's published IP list or reverse DNS. Scrapers borrow GPTBot and Googlebot names.",
    "Look for the expensive paths first: large downloads (zip, PDF, video, images), old URLs that redirect to uncached endpoints, search pages, git history and other dynamic pages. Read the Docs' 73 TB came through an old redirect to an uncached download URL.",
    "Send ETag and Last-Modified headers and answer conditional requests with 304, so a well-behaved crawler only re-downloads what changed. Read the Docs said the crawler that cost it $5,000 didn't use them.",
    "Rate limit per verified bot and per path at your edge, not just per IP. Read the Docs said IP rate limits didn't work because the crawlers came from many addresses.",
    "Use robots.txt for crawlers that honor it, and an edge rule for the rest. Blocking training crawlers doesn't remove you from Google Search; Googlebot and Google-Extended are separate controls.",
    "Compare what each crawler costs with what it sends back: count referred visits per AI platform in your analytics next to its crawl volume, and remember native apps often send no Referer, so referrals are undercounted.",
    "If others need your data in bulk, give them a cheaper channel such as a dump, feed or rate-limited API, and ask them to identify themselves to get higher limits. Wikimedia's rule of thumb is that stronger identification earns a higher limit."
  ],
  "sections": [
    {
      "heading": "Published bandwidth bills from AI crawlers",
      "points": [
        "Read the Docs, July 2024: one crawler pulled 73 TB of zipped HTML in May 2024 and almost 10 TB in a single day, over $5,000 in bandwidth charges. Read the Docs said the crawler had no bandwidth limiting and didn't use ETags or Last-Modified to skip unchanged files; the company was responsive and covered the cost. In June 2024 another 10 TB, mostly zipped HTML and PDFs, came through Facebook's content downloader; Read the Docs' email to the contact in the bot's user agent bounced.",
        "Blocking the bots Cloudflare classifies as AI crawlers cut Read the Docs' download bandwidth by 75%, from about 800 GB a day to about 200 GB a day. It estimated that traffic would cost about $50 a day, or $1,500 a month, if it hit its origin servers.",
        "Wikimedia, April 2025: multimedia download bandwidth grew 50% from January 2024, largely from programs scraping Wikimedia Commons images to feed AI models. At least 65% of the resource-heavy traffic reaching its core datacenters came from bots, against about 35% of pageviews. Its March 2026 follow-up says it now blocks or throttles about 30% of automated requests from crawlers that don't follow its robot policy, up to billions of requests a day, and that some crawlers hide behind residential proxies."
      ]
    },
    {
      "heading": "Why bot traffic costs more per request than human traffic",
      "points": [
        "People cluster on the same popular pages, which stay in the CDN cache. Crawlers bulk-read everything, including old and rarely visited pages, so their requests miss the cache and go to your origin, where you pay for bandwidth and compute. Wikimedia and Read the Docs both describe this.",
        "Large files make it worse. A zip or PDF that few humans download falls out of cache, so each bot fetch is a full origin download, and a buggy crawler can repeat it hundreds of times."
      ]
    },
    {
      "heading": "What AI crawlers send back",
      "points": [
        "Cloudflare's crawl-to-refer ratio counts HTML pages a platform's bots fetch per page visit it refers. For July 2025 Cloudflare reported Anthropic at about 38,000 to 1 (down from about 287,000 to 1 in January; Cloudflare suggests Claude's web search with cited links may be helping), OpenAI about 1,091 to 1, Perplexity about 195 to 1, Microsoft about 41 to 1 and Google about 5.4 to 1. Cloudflare also found training made up nearly 80% of AI bot crawling.",
        "Cloudflare notes these ratios likely overstate the gap, because traffic from native apps usually has no Referer header, so it isn't counted as a referral. They are also averages across Cloudflare's network; your own logs are the number that matters for your bill."
      ]
    }
  ],
  "checks": [],
  "ask_rightshold": {
    "start": "https://www.rightshold.com/start?ref=seo",
    "mcp_endpoint": "https://www.rightshold.com/mcp",
    "ask": "What does my site's robots.txt currently tell GPTBot, ClaudeBot, CCBot, Google-Extended and the AI search and user-fetch bots, so I know which crawlers I've asked to stay away before I set edge limits?",
    "mcp_call": {
      "tool": "read_ai_policy",
      "arguments": {
        "domain": "example.com"
      }
    },
    "also": [
      "Free, no key, cached 6 hours. It reads robots.txt per AI user agent (training, search and user-fetch tokens, including Content-Signal lines), ai.txt, /.well-known/tdmrep.json and whether llms.txt exists, then summarizes what each bot is told.",
      "It shows what you've published, not what bots do. It doesn't read your logs, measure bandwidth, verify IPs or block anything. For crawlers that sign requests with Web Bot Auth, gate_check verifies the signature so you can key rate limits on a verified signer instead of a user-agent string."
    ],
    "scope": "Rightshold reads a site's public AI and crawl signals and checks agent signatures you send it. It doesn't see your traffic, calculate your hosting costs, rate limit or block crawlers. The figures above are from the cited reports for the dates given and won't match any other site's costs."
  },
  "sources": [
    {
      "title": "Read the Docs: AI crawlers need to be more respectful (July 25, 2024)",
      "url": "https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/"
    },
    {
      "title": "Wikimedia Foundation: How crawlers impact the operations of the Wikimedia projects (April 1, 2025)",
      "url": "https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/"
    },
    {
      "title": "Wikimedia Foundation: Quo Vadis, Crawlers? Progress and what's next on safeguarding our infrastructure (March 26, 2026)",
      "url": "https://diff.wikimedia.org/2026/03/26/quo-vadis-crawlers-progress-and-whats-next-on-safeguarding-our-infrastructure/"
    },
    {
      "title": "Cloudflare: The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (2025)",
      "url": "https://blog.cloudflare.com/crawlers-click-ai-bots-training/"
    },
    {
      "title": "Cloudflare: The crawl before the fall of referrals (crawl-to-refer ratio methodology, 2025)",
      "url": "https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/"
    }
  ],
  "related": [
    {
      "title": "How to slow down AI crawlers overloading your server without hurting Google rankings",
      "url": "https://www.rightshold.com/answers/how-to-slow-down-ai-crawlers-overloading-your-server-without-hurting-google-rankings"
    },
    {
      "title": "Why Google Analytics shows almost no AI bot traffic, and how to measure it",
      "url": "https://www.rightshold.com/answers/why-google-analytics-shows-almost-no-ai-bot-traffic-and-how-to-measure-it"
    },
    {
      "title": "How to block AI training crawlers in robots.txt without leaving AI search",
      "url": "https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search"
    },
    {
      "title": "How to charge AI crawlers per request with HTTP 402 without blocking search crawlers",
      "url": "https://www.rightshold.com/answers/how-to-charge-ai-crawlers-per-request-with-http-402-without-blocking-search-crawlers"
    }
  ],
  "computed_from": "Curated guidance; license verdicts computed from Rightshold's license profiles at render time. Not legal advice.",
  "updated": "2026-11-10"
}