Meta runs several crawlers (facebookexternalhit, meta-externalagent, meta-externalfetcher and others). Which ones can a site block in robots.txt without breaking Facebook and Instagram link previews, which ones may ignore it, and did Meta's proposed business terms that let it crawl sites linked to business accounts change any of that?

What should I check, step by step?

1. Search your logs for each token separately: facebookexternalhit, meta-webindexer, meta-externalads, meta-externalagent and meta-externalfetcher. A rule written for one doesn't apply to the others.

2. To opt out of Meta model training, add "User-agent: meta-externalagent" then "Disallow: /". Meta's own example also shows a separate group for meta-externalads.
3. Keep facebookexternalhit allowed unless you're willing to lose the title, description and image when people share your links on Facebook, Instagram or Messenger. It isn't the training crawler.
4. Decide on meta-webindexer on its own: Meta says allowing it helps Meta AI cite and link to your content, so blocking it trades away citations, not training.
5. If you run Meta ads, think before blocking meta-externalads: Meta describes it only as improving advertising and business products and doesn't document what blocking it changes.
6. Don't rely on robots.txt for meta-externalfetcher. Meta says it may bypass robots.txt for user-requested fetches, so use a login, or a server or CDN rule if you accept the trade-offs, for anything it shouldn't read.
7. If you check old robots.txt files, look for tokens Meta no longer lists (such as FacebookBot); a rule on a retired name does nothing for the current crawlers.
8. Allow up to 24 hours for a change, keep a dated copy of each robots.txt version, and watch Meta's Business Help Center for any new version of the paused terms.

What are Meta's crawlers (facebookexternalhit, meta-externalagent, meta-webindexer) used for?

Meta's developer page lists the user agents behind its most common crawlers. facebookexternalhit crawls apps and websites shared on Meta's apps, caching and displaying the title, description and thumbnail. Meta asks sites to allow-list its user-agent strings or, more securely, its IP addresses, and to serve Open Graph tags in the first megabyte of the page.

• Meta-WebIndexer navigates the web to improve Meta AI search result quality; Meta says allowing it in robots.txt helps it cite and link to your content in Meta AI's responses. Meta-ExternalAds crawls for uses such as improving advertising and other business-related products and services. Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly.
• Meta-ExternalFetcher fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI, including helping AI navigate websites to complete tasks for users. The page also says Meta prefers industry-standard robots.txt over non-standard formats such as NoAI tags.

Which Meta crawlers may ignore robots.txt?

Meta says, in plain words, that Meta-ExternalFetcher may bypass robots.txt rules because it performs fetches requested by the user, and that facebookexternalhit might bypass robots.txt when performing security or integrity checks, such as checking for malware. The page doesn't list a bypass for Meta-WebIndexer, Meta-ExternalAds or Meta-ExternalAgent.

• That puts Meta in line with other companies' user-triggered fetchers, which are generally documented as acting for a person rather than crawling, and so not bound by robots.txt in the same way. A Disallow line still records your preference, but if content must not be fetched, only access control stops it.
• Meta also says crawlers may cache robots.txt for up to 24 hours, so a change isn't instant, and its own robots.txt example names only meta-externalagent and meta-externalads. Tokens without a group of their own fall back to your "User-agent: *" rules.

Did Meta's new Commercial Terms let it crawl and train on business websites, and are they in effect?

In early October 2026 Meta notified business users of updated Commercial Terms, due to take effect on October 30. As reported by Nieman Lab on October 15, the draft authorized Meta to use automated software, and to create test credentials, to crawl domains and pages associated with a business's products or services, and to use that content to provide, develop, improve, research, train and maintain Meta products. Business use included running ads, managing a Page and using a product catalogue, with no opt-out other than stopping business use.

• On October 16 Meta removed "train" and limited the use to advertising and business products; a Meta spokesperson said robots.txt preferences are respected and that sites can still disable the crawler with robots.txt. On October 23 Meta moved the effective date to November 20. On November 5 Meta took the preview down and its Business Help Center said the updates to the Commercial Terms, Product Catalog Terms, Self-Serve Ad Terms and Ad Creative Generative AI Terms are paused, won't take effect on November 20, 2026, and that the existing terms remain in effect.
• Nieman Lab also reported preliminary injunctions in Norway and Brazil and a formal investigation by the UK Competition and Markets Authority. For a site owner, the practical position on November 10, 2026: the current terms apply, Meta's crawler page is the description of how its crawlers behave, and robots.txt is the tool Meta points to. If a new version is published, read whether it says anything about robots.txt; none of the versions reported so far did.