# Meta runs several crawlers (facebookexternalhit, meta-externalagent, meta-externalfetcher and others). Which ones can a site block in robots.txt without breaking Facebook and Instagram link previews, which ones may ignore it, and did Meta's proposed business terms that let it crawl sites linked to business accounts change any of that?

**Short answer:** Meta's crawler documentation (updated May 21, 2026) lists five user agents, each with its own robots.txt token. facebookexternalhit builds link previews when your page is shared on Facebook, Instagram or Messenger; Meta-WebIndexer crawls to improve Meta AI search results and helps Meta AI cite and link to you; Meta-ExternalAds crawls to improve advertising and business products; Meta-ExternalAgent crawls for uses such as training foundation AI models; and Meta-ExternalFetcher fetches single links at a user's request, including for AI agents completing tasks. To keep your pages out of Meta's model training, block meta-externalagent; to keep previews, leave facebookexternalhit allowed. Two of the five may not obey a block: Meta says Meta-ExternalFetcher may bypass robots.txt because the fetch was requested by a user, and facebookexternalhit might bypass it for security or integrity checks. Meta asks for up to 24 hours for changes. As for the terms: in October 2026 Meta announced updated Commercial Terms that would have authorized it to crawl sites associated with business accounts, first with "train" among the permitted uses, then revised to remove it. On November 5, 2026 Meta paused the update indefinitely; its Business Help Center says the existing terms remain in effect, and Meta has said throughout that robots.txt preferences are respected, though no published version of the new terms said so.

## Checklist
1. Search your logs for each token separately: facebookexternalhit, meta-webindexer, meta-externalads, meta-externalagent and meta-externalfetcher. A rule written for one doesn't apply to the others.
2. To opt out of Meta model training, add "User-agent: meta-externalagent" then "Disallow: /". Meta's own example also shows a separate group for meta-externalads.
3. Keep facebookexternalhit allowed unless you're willing to lose the title, description and image when people share your links on Facebook, Instagram or Messenger. It isn't the training crawler.
4. Decide on meta-webindexer on its own: Meta says allowing it helps Meta AI cite and link to your content, so blocking it trades away citations, not training.
5. If you run Meta ads, think before blocking meta-externalads: Meta describes it only as improving advertising and business products and doesn't document what blocking it changes.
6. Don't rely on robots.txt for meta-externalfetcher. Meta says it may bypass robots.txt for user-requested fetches, so use a login, or a server or CDN rule if you accept the trade-offs, for anything it shouldn't read.
7. If you check old robots.txt files, look for tokens Meta no longer lists (such as FacebookBot); a rule on a retired name does nothing for the current crawlers.
8. Allow up to 24 hours for a change, keep a dated copy of each robots.txt version, and watch Meta's Business Help Center for any new version of the paused terms.

## What are Meta's crawlers (facebookexternalhit, meta-externalagent, meta-webindexer) used for?
- Meta's developer page lists the user agents behind its most common crawlers. facebookexternalhit crawls apps and websites shared on Meta's apps, caching and displaying the title, description and thumbnail. Meta asks sites to allow-list its user-agent strings or, more securely, its IP addresses, and to serve Open Graph tags in the first megabyte of the page.
- Meta-WebIndexer navigates the web to improve Meta AI search result quality; Meta says allowing it in robots.txt helps it cite and link to your content in Meta AI's responses. Meta-ExternalAds crawls for uses such as improving advertising and other business-related products and services. Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly.
- Meta-ExternalFetcher fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI, including helping AI navigate websites to complete tasks for users. The page also says Meta prefers industry-standard robots.txt over non-standard formats such as NoAI tags.

## Which Meta crawlers may ignore robots.txt?
- Meta says, in plain words, that Meta-ExternalFetcher may bypass robots.txt rules because it performs fetches requested by the user, and that facebookexternalhit might bypass robots.txt when performing security or integrity checks, such as checking for malware. The page doesn't list a bypass for Meta-WebIndexer, Meta-ExternalAds or Meta-ExternalAgent.
- That puts Meta in line with other companies' user-triggered fetchers, which are generally documented as acting for a person rather than crawling, and so not bound by robots.txt in the same way. A Disallow line still records your preference, but if content must not be fetched, only access control stops it.
- Meta also says crawlers may cache robots.txt for up to 24 hours, so a change isn't instant, and its own robots.txt example names only meta-externalagent and meta-externalads. Tokens without a group of their own fall back to your "User-agent: *" rules.

## Did Meta's new Commercial Terms let it crawl and train on business websites, and are they in effect?
- In early October 2026 Meta notified business users of updated Commercial Terms, due to take effect on October 30. As reported by Nieman Lab on October 15, the draft authorized Meta to use automated software, and to create test credentials, to crawl domains and pages associated with a business's products or services, and to use that content to provide, develop, improve, research, train and maintain Meta products. Business use included running ads, managing a Page and using a product catalogue, with no opt-out other than stopping business use.
- On October 16 Meta removed "train" and limited the use to advertising and business products; a Meta spokesperson said robots.txt preferences are respected and that sites can still disable the crawler with robots.txt. On October 23 Meta moved the effective date to November 20. On November 5 Meta took the preview down and its Business Help Center said the updates to the Commercial Terms, Product Catalog Terms, Self-Serve Ad Terms and Ad Creative Generative AI Terms are paused, won't take effect on November 20, 2026, and that the existing terms remain in effect.
- Nieman Lab also reported preliminary injunctions in Norway and Brazil and a formal investigation by the UK Competition and Markets Authority. For a site owner, the practical position on November 10, 2026: the current terms apply, Meta's crawler page is the description of how its crawlers behave, and robots.txt is the tool Meta points to. If a new version is published, read whether it says anything about robots.txt; none of the versions reported so far did.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "Does example.com's robots.txt block meta-externalagent from training on our pages, and which group did it match? Show me the same for the other AI training crawlers so I can see what's still allowed."

```json
{
  "tool": "read_ai_policy",
  "arguments": {
    "domain": "example.com"
  }
}
```
- Free, no key, cached about 6 hours. For 16 AI crawler, search and user-fetch tokens, including meta-externalagent (counted as a training crawler), it reports whether the path is allowed and which group matched, plus Content-Signal lines, ai.txt, W3C TDMRep and whether llms.txt exists. Pass a path to check one section.
- It doesn't check facebookexternalhit, meta-webindexer, meta-externalads or meta-externalfetcher; read those groups in your file yourself. It reports what your files say, not whether Meta honors them or what Meta has already collected.

_Rightshold reads a site's public crawl and AI-use signals. It isn't affiliated with Meta and can't make any crawler honor a preference. Crawler behavior is as described on Meta's crawler page (updated May 21, 2026); the terms timeline is from Meta's statements as reported by Nieman Lab through November 5, 2026, checked on November 10, 2026. Whether a contract can override a robots.txt preference depends on the terms and your jurisdiction. Not legal advice._

## Sources
- [Meta for Developers: Meta Web Crawlers (updated May 21, 2026)](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- [Nieman Lab: Meta's new commercial terms will allow the scraping of news sites, legal experts say (October 15, 2026)](https://www.niemanlab.org/2026/10/metas-new-commercial-terms-will-allow-the-scraping-of-news-sites-legal-experts-say/)
- [Nieman Lab: Legal experts warn Meta's new terms still grant broad scraping permissions, despite language changes (October 17, 2026)](https://www.niemanlab.org/2026/10/legal-experts-warn-metas-new-terms-still-grant-broad-scraping-permissions-despite-language-changes/)
- [Nieman Lab: Meta delays rollout of terms that would allow broad scraping of news sites (October 2026)](https://www.niemanlab.org/2026/10/meta-delays-rollout-of-terms-that-would-allow-broad-scraping-of-news-sites-legal-experts-are-still-skeptical/)
- [Nieman Lab: Meta pulls down controversial terms of service (November 5, 2026)](https://www.niemanlab.org/2026/11/meta-pulls-down-controversial-terms-of-service/)
- [IETF: RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)

## Related
- [How to block AI training crawlers in robots.txt without leaving AI search](https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search)
- [How to tell ChatGPT-User fetches from AI training crawlers in server logs](https://www.rightshold.com/answers/how-to-tell-chatgpt-user-fetches-from-ai-training-crawlers-in-server-logs)
- [Should you block Amazonbot? Amazonbot vs Amzn-SearchBot vs Amzn-User](https://www.rightshold.com/answers/should-you-block-amazonbot-and-what-is-the-difference-between-amazonbot-amzn-searchbot-and-amzn-user)
- [Why an AI training opt-out or robots.txt is not access control](https://www.rightshold.com/answers/why-an-ai-training-opt-out-or-robots-txt-is-not-access-control)

_Not legal advice. Updated 2026-11-10. JSON: https://www.rightshold.com/answers/which-meta-crawlers-can-you-block-in-robots-txt-and-did-metas-new-business-terms-change-that.json_