How to Allow GPTBot, PerplexityBot and ClaudeBot in robots.txt

After reading about AI search, the instinct is to open robots.txt and "add the AI bots." It is a reasonable instinct that runs into two problems straight away.

The first: robots.txt only ever says no. There is no line you can write that grants a crawler access it did not already have. Every crawler is allowed everywhere by default, and the file's entire job is to carve out exceptions. So "allowing GPTBot" really means one of two things: making sure nothing in your file is blocking it, and deciding whether you actually want it through at all.

The second: the three bots in the title do three different jobs, and only some of them affect whether an AI engine can cite you. GPTBot brings you nothing in a ChatGPT answer. A bot most people have never heard of does.

This guide covers the current user-agent tokens for every major AI crawler, which ones to let through and why, a robots.txt file you can copy, and the handful of mechanics that cause sites to block AI crawlers by accident.

robots.txt allows by default

Google's own robots.txt specification puts it plainly: "By default, there are no restrictions for crawling." If you have no robots.txt file, every crawler is welcome on every page. If you have one but nothing in it matches a given crawler, that crawler is still welcome.

This changes what the task actually is. You are not adding crawlers to an allowlist. You are:

1. Checking that no existing rule blocks the crawlers you want. An old Disallow: / from a staging setup, a broad rule from a migration, a security tool's default. These are what keep AI crawlers out, not the absence of a friendly line.
2. Deciding which crawlers, if any, you want to keep out, and blocking only those, precisely, without catching the ones that bring you AI citations.

An explicit Allow line still has a use, covered further down, but it is a safety measure against your own broader rules, not a permission slip.

Which token sits in which bucket

AI crawlers do three different jobs, and our guide to AI crawler reachability explains the categories in full. The short version, because it decides what a block costs: training crawlers feed a future model and bring you nothing today, retrieval crawlers build the index an engine cites from, and user-triggered fetchers run only when a real person points the assistant at your page.

For a robots.txt decision, all that matters is sorting each real token into the right bucket:

Allow the top two rows. Treat the third as a separate decision with nothing to do with AI visibility. And of the files at your site root, robots.txt is the one AI systems genuinely read, so the choice you make here outweighs anything in an llms.txt.

The current bots, by operator

Tokens change. These are current as of this writing, taken from each operator's own crawler documentation, which is the only place worth trusting since third-party lists go stale fast.

OpenAI (ChatGPT)

From OpenAI's crawler documentation:

Perplexity

From Perplexity's bot documentation:

Getting a page indexed is only half the job with Perplexity, and our guide to getting cited by Perplexity covers the content side once the crawler is through.

Anthropic (Claude)

From Anthropic's crawler documentation:

Anthropic's older tokens, Claude-Web and anthropic-ai, are retired. Rules naming them do nothing now, though leaving them in place does no harm either.

Google

Google is a special case because its AI features are not separate products with their own crawlers. AI Overviews and AI Mode are part of Google Search, built from the same index that ordinary results come from, crawled by the same Googlebot.

• Googlebot must be allowed. Blocking it removes you from Google Search entirely, AI Overviews included. There is no token that keeps you in the blue links but out of the AI answer.
• Google-Extended is a training-control token, not a crawler. It "doesn't have a separate HTTP request user agent string." Disallowing it stops your content being used to train future Gemini models, and Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Both facts are on Google's crawler documentation.
• Google-CloudVertexBot only fetches your site if you, or someone you have authorised, are building a Vertex AI agent that uses your pages. Most sites will never see it and do not need a rule for it.

The rest, briefly

• bingbot has no AI-specific token. It builds Bing's index, which Microsoft Copilot draws on and which ChatGPT search has historically leaned on as a retrieval layer. Allow it, and confirm you are actually indexed in Bing, which we cover in the ChatGPT citation guide.
• Applebot feeds Apple's search and Siri; Applebot-Extended is Apple's training opt-out token, same pattern as Google-Extended.
• Amazonbot, Meta-ExternalAgent, CCBot (Common Crawl), and Bytespider (ByteDance) are training crawlers. CCBot and Bytespider in particular have a mixed record on actually obeying robots.txt, so a rule for them is a request, not a guarantee.

A robots.txt that lets the retrieval crawlers through

If your file is currently empty or absent and you just want AI engines to be able to cite you, you do not need to write anything. Default access already covers it. Confirm your file does not block them and stop there.

If you have a restrictive User-agent: * rule that you need for other reasons, this is where an explicit block per crawler earns its place. Here is a file that keeps a general restriction but lets the AI retrieval and user crawlers through:

# General rule for everything not named below
User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/

# AI retrieval crawlers: allow fully so they can cite you
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# AI user-triggered fetchers: allow so a person's question can reach the page
User-agent: ChatGPT-User
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-User
Allow: /

# AI training crawlers: block these if you want to, it does not affect citations
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Two things about that file matter more than they look.

Each named crawler follows only its own group. Google's spec is explicit: "User agent specific groups and global groups (*) are not combined." So OAI-SearchBot reads its own three lines and ignores the User-agent: * block entirely. That is why the explicit Allow: / blocks work as a rescue: even if your * rule said Disallow: /, a crawler with its own named group never sees it.

Leave the training-crawler blocks out if you have not decided. Blocking GPTBot and ClaudeBot is a real choice with real trade-offs, not a default. If you are not sure, do not block them, and revisit it deliberately later.

If your existing file is short and you only want to be sure the retrieval crawlers are not caught by anything, the minimal version is just the explicit allows:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Those three groups override whatever your User-agent: * block says, for those three crawlers only, and leave everything else untouched.

If you are on WordPress

WordPress does not ship a physical robots.txt file. It serves a virtual one, generated on each request, and by default that virtual file is permissive. The moment a real robots.txt file exists in your site root, WordPress stops generating its own and serves the file instead, so an old file left behind by a host migration or a former developer can quietly be the thing in charge.

Most SEO plugins add their own editor for it. Yoast, Rank Math, and All in One SEO each have an "edit robots.txt" screen, some editing the virtual file, some writing a real one to your root. If two of them are active, or a plugin is editing the virtual file while a physical file also exists, only one version is actually served, and it is not always the one you last edited. Check yourdomain.com/robots.txt in a browser to see which content won, then make your changes in whichever tool produced it.

The mistakes that block AI crawlers by accident

A leftover Disallow: / under User-agent: *. Written to keep a staging site out of search, or during a scraper incident, and never removed. It blocks every crawler that does not have its own named group, which is most of them. This is the single most common cause of a site being invisible to AI engines.

Assuming a crawler obeys both * and its own rule. It does not. If you add User-agent: GPTBot with a single Disallow: /some-path line, GPTBot now follows only that group. Your careful User-agent: * restrictions no longer apply to it at all. Name a crawler and you take on responsibility for its whole ruleset.

A Disallow path that catches more than you meant. Disallow: /app also blocks /applications, /apple-pay, and /approach, because it is a prefix match, not a folder match. Use a trailing slash (/app/) when you mean a folder.

robots.txt on the wrong host. The file is per host, per protocol, and per port. https://example.com/robots.txt does not govern https://www.example.com/ or https://blog.example.com/. Each hostname a crawler can reach needs its own file.

A typo in the token. User-agent: OAI-Searchbot with the wrong casing is fine, matching is case-insensitive, but OAI-Search-Bot or OpenAI-SearchBot matches nothing and silently does nothing.

A blanket block aimed at SEO scrapers. Rules that Disallow: / for AhrefsBot, SemrushBot, and a long tail of others sometimes get extended with a catch-all that sweeps in AI retrieval crawlers too. Check that list against the tokens above.

robots.txt is a request, not a wall

For allowing the crawlers you want, robots.txt is reliable. OpenAI, Anthropic, and Perplexity all state in writing that their indexing crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) honour it, and Google's have for decades. Allow those and they come in.

For blocking, it is only as good as the crawler's cooperation, and two things sit outside your file's control:

A crawler can ignore it. Every user-triggered fetcher is exempt by design, not by misbehaviour: when a person hands your URL to an assistant, the fetch is treated as that person visiting, and robots.txt governs automated crawling, not visits. That is why Perplexity-User and ChatGPT-User reach a page even when the matching crawler is blocked. Separately, Bytespider has been caught crawling sites that disallowed it, and CCBot applies changes on its own slow schedule. A block on a crawler that will not cooperate needs an IP or WAF rule, not a text line.

Your CDN or firewall can block a crawler your robots.txt welcomes. Cloudflare and similar services have shipped "block AI bots" controls that are on by default on some plans, and broad bot rules catch AI crawlers as collateral. Your file can say "come in" while your edge returns a 403. After fixing robots.txt, check your bot-management settings, then your access logs. Our guide to AI crawler reachability goes deeper on this layer.

Check your own robots.txt

Three checks, specific to the file itself:

• Read what is actually deployed. Open https://yourdomain.com/robots.txt in a browser rather than trusting a copy in your repo, and do it for www, non-www, and any subdomain that serves real content, since each hostname has its own file.
• Trace one page through your rules, per token. Pick an important URL and walk the file top to bottom as OAI-SearchBot, then PerplexityBot, then Claude-SearchBot, remembering that each one reads only its own named group or the * group, never both. Google Search Console's robots.txt report does this automatically, but only for Googlebot.
• Confirm no named group is stricter than you think. The moment you add a line for GPTBot, your User-agent: * rules stop applying to it. Read every named group as a complete, standalone ruleset.

Once the file is right, the confirmation is in your access logs: within a few weeks the retrieval crawlers should start showing up and returning periodically. If they never do, the block is above robots.txt, in your CDN or host. Our free audit checks the file against the current crawler list and flags anything keeping the retrieval bots out.

Want to know exactly which AI crawlers your site is letting in and which it is quietly blocking? Run a free AI readiness check for a robots.txt audit against the current crawler list, plus the crawler-versus-browser comparison that catches what a bot really sees.