If I block Common Crawl's CCBot in robots.txt, does that take my site out of Common Crawl and the AI models trained on it, or only stop future crawls?

What should I check, step by step?

1. Add a CCBot group to robots.txt: "User-agent: CCBot" then "Disallow: /". Common Crawl says this stops the crawler, and it re-checks robots.txt periodically. If you only want it slower, it honours Crawl-delay instead.

2. Add separate lines for the other training crawlers you want to keep out (for example GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent). A CCBot rule only applies to CCBot.
3. Make sure robots.txt itself loads: a 200 with plain text at /robots.txt on every host you serve (www and the bare domain), not a login page, a redirect loop or a firewall challenge.
4. Check your logs for fake CCBot traffic. Common Crawl says other crawlers falsely use its name. Real CCBot comes from the ranges in index.commoncrawl.org/ccbot.json, and IPv4 addresses reverse-resolve to *.crawl.commoncrawl.org and forward-resolve back to the same IP. Treat anything else claiming to be CCBot as a scraper.
5. To see whether your pages were captured, query the CDX index for a recent crawl (index.commoncrawl.org lists them; the newest is CC-MAIN-2026-43, October 2026) with url=yourdomain.com/* and output=json. Go slowly: the API is heavily rate limited and blocks clients that send too many requests.
6. For content already collected, send a removal request to info@commoncrawl.org naming your domains and URLs, and keep the reply. Common Crawl describes removal as multi-step and says its index tools aren't a legal confirmation of removal, so ask what it has done and when.
7. Contact downstream datasets separately if it matters to you. FineWeb's dataset card links a removal form for personal information and its changelog records domains removed after a cease-and-desist notice; other datasets have their own processes or none.
8. Don't rely on JavaScript to hide paid or private text. CCBot doesn't execute JavaScript, so if the full text is in the HTML and only hidden by a script, the crawler receives the full text. Serve restricted content only after login.
9. Keep a dated copy of your robots.txt each time you change it, so you can show when you opted out if a dispute ever turns on timing.

Does blocking CCBot in robots.txt stop Common Crawl from crawling my site?

CCBot identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/). Common Crawl's FAQ says it checks robots.txt first and fetches a page only if crawling is allowed, follows Crawl-delay, honours nofollow on your links, supports conditional GET, and backs off when a site returns 429 or 5xx errors.

• Each month Common Crawl crawls several billion new pages from a frontier of over a trillion, and it says it archives a randomly selected subset of each site rather than whole sites. A robots.txt block applies to the crawls after CCBot next reads the file.
• Common Crawl warns that other crawlers falsely identify themselves as CCBot. It now crawls from dedicated IP ranges with reverse DNS (not yet over IPv6), published as JSON at index.commoncrawl.org/ccbot.json; the list dated August 11, 2026 has one IPv6 /56 and four small IPv4 blocks. A request from outside those ranges isn't Common Crawl, whatever its user agent says, and blocking it is between you and whoever sent it.

Does Common Crawl delete pages it already archived when you opt out?

In a November 4, 2025 statement, Common Crawl's executive director Rich Skrenta said its archives are stored in an immutable format (WARC files) that it can't edit after publication without breaking their integrity, so in response to removal requests it removes or filters affected URLs from subsequent crawls and makes them inaccessible through its public tools and indices. He added that a "no captures" result in its search tools reflects how indices are generated, not what is stored internally, and that no one at Common Crawl has claimed the work is instantaneous or complete.

• That statement responded to an Atlantic article the same day ("The Nonprofit Doing the AI Industry's Dirty Work") that said Common Crawl's archives held paywalled news articles and that content publishers had asked it to remove was still there. Press Gazette reported that content from The New York Times and from publishers represented by the Danish Rights Alliance, which Common Crawl had agreed to remove, was still available according to The Atlantic. Common Crawl called the article's claims false and misleading.
• On September 17, 2025, Common Crawl began publishing an Opt-Out Registry of the legal opt-out requests it has received, to warn its downstream users. Press Gazette reported in June 2026 that it lists publishers including the BBC, The Guardian, the Financial Times, The Washington Post, Reuters and the Associated Press, plus more than 900 news sites under an entry from the News/Media Alliance, and that the trade body Digital Content Next had sent Common Crawl a cease-and-desist letter asking it to stop collecting members' content and remove what it already holds.

Does removing my site from Common Crawl remove it from AI training datasets like C4 or FineWeb?

Most AI builders don't train on raw Common Crawl. Mozilla Foundation's February 2024 report found that at least 30 of 47 text-generation models released between 2019 and October 2023 (64%) used at least one filtered version of it, usually one published by someone else, such as Google's C4 or EleutherAI's Pile-CC. It also noted that filtered Common Crawl made up more than 80% of the tokens in OpenAI's GPT-3 training data.

• Those filtered versions are copies. Hugging Face's FineWeb, for example, is built from every Common Crawl snapshot since 2013 and lists the source WARC file for each document. Its changelog shows that in January 2025 it removed specific domains in response to a cease-and-desist notice, so changes at one dataset don't flow automatically to the others, or back to Common Crawl.
• A model that has already been trained doesn't forget your pages when the source dataset changes. An opt-out today affects which future crawls and dataset builds include you.