meta-externalagent: what it is and how to allow or block it
Meta’s crawler for uses including training its AI models.
Last checked 2026-10-08
- Company
- Meta
- Used for
- Meta AI model training and indexing
- Job
- Collects pages to train AI models
- Follows robots.txt
- Yes
- Name in robots.txt
- meta-externalagent
What meta-externalagent does
Meta says this crawler “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”
Meta runs several other crawlers with their own names. Meta-WebIndexer is the one Meta says helps it “cite and link to your content in Meta AI’s responses.” facebookexternalhit builds link previews when your page is shared on one of Meta’s apps. Each is named separately in robots.txt, so blocking meta-externalagent does not block the others.
Should you allow it?
A genuine choice about training. Meta names training and indexing together for this bot, so blocking it is broader than a pure training opt-out, but Meta AI’s citations are handled by the separate Meta-WebIndexer.
What blocking it changes: Your pages should no longer be collected by this crawler, including for Meta’s AI training.
Allow or block meta-externalagent in robots.txt
Meta’s documentation uses robots.txt rules to control it, and notes that changes can take up to 24 hours because crawlers may keep a copy of your robots.txt for that long.
To allow it
User-agent: meta-externalagent Allow: /
To block it
User-agent: meta-externalagent Disallow: /
robots.txt lets every bot in by default, so you only need the allow lines if another rule in your file would otherwise block it. Our robots.txt guide explains how rules for different bots combine.
How it appears in your server logs
Meta publishes this as the bot’s user-agent string, the label it sends with every visit:
meta-externalagent/1.1
How to check a visit is really meta-externalagent
Meta’s crawler documentation does not publish an IP list for this bot, so a visit can only be judged by its user-agent string, which anyone can copy.
Our guide to server log checks shows where to find these visits on common hosting setups.
Source: Meta crawler documentation, checked 2026-10-08. Bots and their rules change often, so check the company’s own page before making a decision that matters.
Other Meta bots
See which AI bots your site lets in
The free AI readiness check reads your robots.txt and reports which of the main AI crawlers are allowed or blocked, along with everything else that decides whether AI can understand your site.
Check your site free