Googlebot: what it is and how to allow or block it
Google’s main search crawler. Its index also feeds AI Overviews and AI Mode.
Last checked 2026-10-08
- Company
- Used for
- Google Search, including AI Overviews and AI Mode
- Job
- Search engine crawler that also feeds AI
- Follows robots.txt
- Yes
- Name in robots.txt
- Googlebot
What Googlebot does
Googlebot builds Google’s search index. Google says the crawl preferences you set for it apply to “Google Search (including Discover and all Google Search features)”.
AI Overviews and AI Mode are features of Google Search, so they draw on what Googlebot collects. There is no separate robots.txt name that keeps you in normal results but out of the AI answers.
Should you allow it?
Allow it. To limit what Google’s AI features can quote, use the snippet controls covered in our AI Overviews opt-out guide instead of blocking the crawler.
What blocking it changes: Your pages drop out of Google Search entirely, which also takes them out of AI Overviews and AI Mode. For almost every business that is far too high a price.
Allow or block Googlebot in robots.txt
Googlebot follows robots.txt. The user-agent string above is the desktop version; Google replaces W.X.Y.Z with the current Chrome version, and also crawls with a smartphone version.
To allow it
User-agent: Googlebot Allow: /
To block it
User-agent: Googlebot Disallow: /
robots.txt lets every bot in by default, so you only need the allow lines if another rule in your file would otherwise block it. Our robots.txt guide explains how rules for different bots combine.
How it appears in your server logs
Google publishes this as the bot’s user-agent string, the label it sends with every visit:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
How to check a visit is really Googlebot
Google warns that the user-agent string “can be spoofed.” A real Googlebot visit reverse-resolves to a googlebot.com hostname, and Google publishes its crawler IP ranges.
Our guide to server log checks shows where to find these visits on common hosting setups.
Source: Google crawler documentation, checked 2026-10-08. Bots and their rules change often, so check the company’s own page before making a decision that matters.
What actually controls how Google’s AI features use your pages is covered in our AI Overviews opt-out guide.
Other Google bots
See which AI bots your site lets in
The free AI readiness check reads your robots.txt and reports which of the main AI crawlers are allowed or blocked, along with everything else that decides whether AI can understand your site.
Check your site free