meta-webindexer: what it is and how to allow or block it
Meta’s crawler for Meta AI search. It helps Meta AI cite and link to your pages.
Last checked 2026-10-08
- Company
- Meta
- Used for
- Meta AI search
- Job
- Builds an AI search index
- Follows robots.txt
- Yes
- Name in robots.txt
- meta-webindexer
What meta-webindexer does
Meta says Meta-WebIndexer “navigates the web to improve Meta AI search result quality for users.”
It is the Meta crawler that matters for being cited: Meta says that allowing it in robots.txt “helps us cite and link to your content in Meta AI’s responses.” It is separate from meta-externalagent, the crawler Meta names for training.
Should you allow it?
Allow it if you want Meta AI to be able to cite you. You can still block meta-externalagent separately if you want to opt out of training.
What blocking it changes: Meta AI should stop indexing your pages, which makes it less able to cite and link to you in its answers.
Allow or block meta-webindexer in robots.txt
Meta says to block its crawlers by adding a disallow rule for the relevant one in robots.txt, and to allow up to 24 hours for changes to take effect.
To allow it
User-agent: meta-webindexer Allow: /
To block it
User-agent: meta-webindexer Disallow: /
robots.txt lets every bot in by default, so you only need the allow lines if another rule in your file would otherwise block it. Our robots.txt guide explains how rules for different bots combine.
How it appears in your server logs
Meta publishes this as the bot’s user-agent string, the label it sends with every visit:
meta-webindexer/1.1
How to check a visit is really meta-webindexer
Meta’s crawler documentation does not publish an IP list for this bot, so a visit can only be judged by its user-agent string, which anyone can copy.
Our guide to server log checks shows where to find these visits on common hosting setups.
Source: Meta crawler documentation, checked 2026-10-08. Bots and their rules change often, so check the company’s own page before making a decision that matters.
Other Meta bots
See which AI bots your site lets in
The free AI readiness check reads your robots.txt and reports which of the main AI crawlers are allowed or blocked, along with everything else that decides whether AI can understand your site.
Check your site free