Claude-SearchBot: what it is and how to allow or block it
Anthropic’s crawler for Claude’s search results. It decides whether Claude can find your pages.
Last checked 2026-10-08
- Company
- Anthropic
- Used for
- Claude web search
- Job
- Builds an AI search index
- Follows robots.txt
- Yes
- Name in robots.txt
- Claude-SearchBot
What Claude-SearchBot does
Claude-SearchBot reads pages to improve the search results Claude gives its users. Anthropic says it “navigates the web to improve search result quality for users.”
Should you allow it?
Allow it if you want Claude to be able to find and cite your site.
What blocking it changes: Anthropic says blocking it “prevents our system from indexing your content for search optimization,” which “may reduce your site’s visibility and accuracy in user search results.”
Allow or block Claude-SearchBot in robots.txt
Anthropic says its bots honor robots.txt.
To allow it
User-agent: Claude-SearchBot Allow: /
To block it
User-agent: Claude-SearchBot Disallow: /
robots.txt lets every bot in by default, so you only need the allow lines if another rule in your file would otherwise block it. Our robots.txt guide explains how rules for different bots combine.
How to check a visit is really Claude-SearchBot
Anthropic publishes one IP list for its bots.
Our guide to server log checks shows where to find these visits on common hosting setups.
Source: Anthropic crawler documentation, checked 2026-10-08. Bots and their rules change often, so check the company’s own page before making a decision that matters.
See which AI bots your site lets in
The free AI readiness check reads your robots.txt and reports which of the main AI crawlers are allowed or blocked, along with everything else that decides whether AI can understand your site.
Check your site free