# If I block Common Crawl's CCBot in robots.txt, does that take my site out of Common Crawl and the AI models trained on it, or only stop future crawls?

**Short answer:** Only going forward. Common Crawl says CCBot checks robots.txt before fetching, and that "User-agent: CCBot" with "Disallow: /" stops it crawling your site; it keeps re-checking the file, so the rule takes effect on later crawls. It doesn't touch what's already been collected. Common Crawl's archives are stored as WARC files that it says it can't edit after publication, so when a publisher asks for removal it filters those URLs out of later crawls and makes them inaccessible through its public tools and indices; it also says this work isn't instantaneous. Legal removal requests go to info@commoncrawl.org, and since September 17, 2025 Common Crawl has published an Opt-Out Registry listing the legal requests it has received. Neither step reaches copies made before you opted out: filtered datasets built from Common Crawl, such as C4, Pile-CC and Hugging Face's FineWeb, are separate copies with their own removal processes, and models already trained on that data don't change. Blocking CCBot also doesn't block GPTBot, ClaudeBot, Google-Extended, Applebot-Extended or any other crawler; each one needs its own line. So block CCBot now if you don't want to be in future crawls, and treat anything already archived as out of your hands unless you pursue removal with each dataset.

## Checklist
1. Add a CCBot group to robots.txt: "User-agent: CCBot" then "Disallow: /". Common Crawl says this stops the crawler, and it re-checks robots.txt periodically. If you only want it slower, it honours Crawl-delay instead.
2. Add separate lines for the other training crawlers you want to keep out (for example GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent). A CCBot rule only applies to CCBot.
3. Make sure robots.txt itself loads: a 200 with plain text at /robots.txt on every host you serve (www and the bare domain), not a login page, a redirect loop or a firewall challenge.
4. Check your logs for fake CCBot traffic. Common Crawl says other crawlers falsely use its name. Real CCBot comes from the ranges in index.commoncrawl.org/ccbot.json, and IPv4 addresses reverse-resolve to *.crawl.commoncrawl.org and forward-resolve back to the same IP. Treat anything else claiming to be CCBot as a scraper.
5. To see whether your pages were captured, query the CDX index for a recent crawl (index.commoncrawl.org lists them; the newest is CC-MAIN-2026-43, October 2026) with url=yourdomain.com/* and output=json. Go slowly: the API is heavily rate limited and blocks clients that send too many requests.
6. For content already collected, send a removal request to info@commoncrawl.org naming your domains and URLs, and keep the reply. Common Crawl describes removal as multi-step and says its index tools aren't a legal confirmation of removal, so ask what it has done and when.
7. Contact downstream datasets separately if it matters to you. FineWeb's dataset card links a removal form for personal information and its changelog records domains removed after a cease-and-desist notice; other datasets have their own processes or none.
8. Don't rely on JavaScript to hide paid or private text. CCBot doesn't execute JavaScript, so if the full text is in the HTML and only hidden by a script, the crawler receives the full text. Serve restricted content only after login.
9. Keep a dated copy of your robots.txt each time you change it, so you can show when you opted out if a dispute ever turns on timing.

## Does blocking CCBot in robots.txt stop Common Crawl from crawling my site?
- CCBot identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/). Common Crawl's FAQ says it checks robots.txt first and fetches a page only if crawling is allowed, follows Crawl-delay, honours nofollow on your links, supports conditional GET, and backs off when a site returns 429 or 5xx errors.
- Each month Common Crawl crawls several billion new pages from a frontier of over a trillion, and it says it archives a randomly selected subset of each site rather than whole sites. A robots.txt block applies to the crawls after CCBot next reads the file.
- Common Crawl warns that other crawlers falsely identify themselves as CCBot. It now crawls from dedicated IP ranges with reverse DNS (not yet over IPv6), published as JSON at index.commoncrawl.org/ccbot.json; the list dated August 11, 2026 has one IPv6 /56 and four small IPv4 blocks. A request from outside those ranges isn't Common Crawl, whatever its user agent says, and blocking it is between you and whoever sent it.

## Does Common Crawl delete pages it already archived when you opt out?
- In a November 4, 2025 statement, Common Crawl's executive director Rich Skrenta said its archives are stored in an immutable format (WARC files) that it can't edit after publication without breaking their integrity, so in response to removal requests it removes or filters affected URLs from subsequent crawls and makes them inaccessible through its public tools and indices. He added that a "no captures" result in its search tools reflects how indices are generated, not what is stored internally, and that no one at Common Crawl has claimed the work is instantaneous or complete.
- That statement responded to an Atlantic article the same day ("The Nonprofit Doing the AI Industry's Dirty Work") that said Common Crawl's archives held paywalled news articles and that content publishers had asked it to remove was still there. Press Gazette reported that content from The New York Times and from publishers represented by the Danish Rights Alliance, which Common Crawl had agreed to remove, was still available according to The Atlantic. Common Crawl called the article's claims false and misleading.
- On September 17, 2025, Common Crawl began publishing an Opt-Out Registry of the legal opt-out requests it has received, to warn its downstream users. Press Gazette reported in June 2026 that it lists publishers including the BBC, The Guardian, the Financial Times, The Washington Post, Reuters and the Associated Press, plus more than 900 news sites under an entry from the News/Media Alliance, and that the trade body Digital Content Next had sent Common Crawl a cease-and-desist letter asking it to stop collecting members' content and remove what it already holds.

## Does removing my site from Common Crawl remove it from AI training datasets like C4 or FineWeb?
- Most AI builders don't train on raw Common Crawl. Mozilla Foundation's February 2024 report found that at least 30 of 47 text-generation models released between 2019 and October 2023 (64%) used at least one filtered version of it, usually one published by someone else, such as Google's C4 or EleutherAI's Pile-CC. It also noted that filtered Common Crawl made up more than 80% of the tokens in OpenAI's GPT-3 training data.
- Those filtered versions are copies. Hugging Face's FineWeb, for example, is built from every Common Crawl snapshot since 2013 and lists the source WARC file for each document. Its changelog shows that in January 2025 it removed specific domains in response to a cease-and-desist notice, so changes at one dataset don't flow automatically to the others, or back to Common Crawl.
- A model that has already been trained doesn't forget your pages when the source dataset changes. An opt-out today affects which future crawls and dataset builds include you.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "What does example.com's robots.txt tell CCBot and the other AI training crawlers right now? Is CCBot actually disallowed on every path, or only some, and which training crawlers are still allowed?"

```json
{
  "tool": "read_ai_policy",
  "arguments": {
    "domain": "example.com"
  }
}
```
- Free, no key, cached 6 hours. It reads robots.txt for 14 AI crawlers and user-fetch agents, including CCBot, GPTBot, ClaudeBot and Google-Extended, plus ai.txt and W3C TDMRep, and summarizes whether AI training is opted out. Pass a path to check a specific section.
- It reports what your files say, not what Common Crawl has archived. It doesn't query Common Crawl's index, send removal requests or verify crawler IP addresses.

_Rightshold reads a site's public crawl and AI-use signals. It isn't affiliated with Common Crawl or Hugging Face, doesn't see your traffic, and can't remove content from any archive, dataset or model. Common Crawl's practices are as described in its FAQ and statements cited here and may have changed; the disputed points between Common Crawl, The Atlantic and publishers are reported as each side stated them. Not legal advice._

## Sources
- [Common Crawl: FAQ (CCBot, robots.txt, Crawl-delay, JavaScript, IP ranges)](https://commoncrawl.org/faq)
- [Common Crawl: CCBot (blocking, fake CCBot warning, reverse DNS)](https://commoncrawl.org/ccbot)
- [Common Crawl: CCBot IP ranges as JSON](https://index.commoncrawl.org/ccbot.json)
- [Common Crawl: Common Crawl Foundation Opt-Out Registry (September 17, 2025)](https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry)
- [Common Crawl: Setting the Record Straight (November 4, 2025)](https://commoncrawl.org/blog/setting-the-record-straight-common-crawls-commitment-to-transparency-fair-use-and-the-public-good)
- [Press Gazette: US publishers tell Common Crawl to stop scraping and delete archive (June 9, 2026)](https://pressgazette.co.uk/media_law/common-crawl-ai-news-publishers-scraping-cease-and-desist-letter/)
- [Mozilla Foundation: Training Data for the Price of a Sandwich (February 2024)](https://www.mozillafoundation.org/en/research/library/generative-ai-training-data/common-crawl/)
- [Hugging Face: FineWeb dataset card (sources, changelog, removal form)](https://huggingface.co/datasets/HuggingFaceFW/fineweb)

## Related
- [How to block AI training crawlers in robots.txt without leaving AI search](https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search)
- [Why an AI training opt-out or robots.txt is not access control](https://www.rightshold.com/answers/why-an-ai-training-opt-out-or-robots-txt-is-not-access-control)
- [How to catch crawlers that ignore robots.txt with a disallowed trap URL](https://www.rightshold.com/answers/how-to-catch-crawlers-that-ignore-robots-txt-with-a-disallowed-trap-url)
- [Does a no-AI notice on your art do anything legally?](https://www.rightshold.com/answers/does-a-no-ai-notice-on-your-art-do-anything-legally)

_Not legal advice. Updated 2026-11-10. JSON: https://www.rightshold.com/answers/does-blocking-ccbot-in-robots-txt-remove-your-site-from-common-crawl-and-ai-training-data.json_