ClaudeBot: what it is and how to allow or block it
Anthropic’s crawler for collecting pages that may be used to train Claude models.
Last checked 2026-10-08
- Company
- Anthropic
- Used for
- Anthropic model training
- Job
- Collects pages to train AI models
- Follows robots.txt
- Yes
- Name in robots.txt
- ClaudeBot
What ClaudeBot does
ClaudeBot gathers pages for Anthropic’s model training. Anthropic describes it as helping “enhance the utility and safety of our generative AI models.”
It is separate from the bots that let Claude search the web or open a page for a user. Those are Claude-SearchBot and Claude-User.
Should you allow it?
A genuine choice about training, like GPTBot. Blocking it does not remove you from Claude’s search results, which Claude-SearchBot handles.
What blocking it changes: In Anthropic’s words, it “signals that the site’s future materials should be excluded from our AI model training datasets.” It does not affect Claude’s search or user-requested visits.
Allow or block ClaudeBot in robots.txt
Anthropic says its bots honor robots.txt and also support the non-standard Crawl-delay setting, so you can slow ClaudeBot down instead of blocking it.
To allow it
User-agent: ClaudeBot Allow: /
To block it
User-agent: ClaudeBot Disallow: /
robots.txt lets every bot in by default, so you only need the allow lines if another rule in your file would otherwise block it. Our robots.txt guide explains how rules for different bots combine.
How to check a visit is really ClaudeBot
Anthropic publishes the IP addresses its bots use. Anthropic also warns that blocking by IP address alone can stop its bots from reading your robots.txt, so robots.txt is the more reliable way to opt out.
Our guide to server log checks shows where to find these visits on common hosting setups.
Source: Anthropic crawler documentation, checked 2026-10-08. Bots and their rules change often, so check the company’s own page before making a decision that matters.
See which AI bots your site lets in
The free AI readiness check reads your robots.txt and reports which of the main AI crawlers are allowed or blocked, along with everything else that decides whether AI can understand your site.
Check your site free