How to Allow AI Crawlers Through Cloudflare and Other Firewalls

Your robots.txt says yes to ChatGPT, Perplexity and Claude. Your firewall says no. Nothing on your site tells you which one wins, and the answer is usually the firewall, because it gets to the visitor first.

This is one of the quietest ways a site disappears from AI answers. The page is fine, the schema is fine, the robots.txt file is fine. But a security setting switched on to stop scrapers is turning away the crawlers that would have quoted you, and it does it before your site ever sees the request. No error shows up in your dashboard. No warning arrives by email. The AI engines just stop coming back.

This guide is about finding that block and opening it, one service at a time: Cloudflare, AWS, Vercel, WordPress security plugins, and the big enterprise bot managers. But it starts with the part most guides skip, which is why you can't find this problem by testing your own site from the outside.

Why you can't test this from your own computer

The natural instinct is to pretend to be the crawler. Send a request to your homepage with GPTBot as the user agent, the name a visitor gives itself, and see whether it gets through. Plenty of online "is my site blocking AI bots" testers do exactly that.

It doesn't work, for a simple reason: modern firewalls don't decide by name. Anyone can type GPTBot into a request. So a good firewall checks the name against where the request actually came from. OpenAI, Anthropic, Perplexity and Google each publish the network addresses their crawlers use, and the firewall checks whether the visit came from one of them. A request from your laptop claiming to be GPTBot fails that check. It gets treated as a fake, which tells you nothing about how the real GPTBot is treated.

The opposite mistake happens too. Your firewall might let your fake-GPTBot request through, because it only blocks bots it has verified as AI bots, and your laptop isn't one. You see a clean result while the real crawler is being refused.

The same limit applies to us. Our own AI readiness check visits your site as itself, under its own name, because sending requests under another company's crawler name from our servers would be dishonest and would rightly get us blocked. It reads your robots.txt against every AI crawler we track, and it notices when a page refuses to load at all. What it cannot see is a firewall rule aimed only at the real GPTBot or the real PerplexityBot.

So there is exactly one reliable test: the real crawler's real visit, recorded by the service that answered it. Every firewall in this guide keeps that record. The work is knowing where it is and what to look for.

First, decide which bots you want through

Not every AI crawler does the same job, and you don't have to let them all in. The ones that matter for being quoted fall into two groups.

Search crawlers build the index an AI engine searches when someone asks a question. If these can't reach you, you can't be cited. The main ones are OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, and Googlebot, which feeds Google's AI Overviews and AI Mode as well as normal search.

User-triggered fetchers visit when a person asks an assistant about a specific page, or pastes your link into a chat. These are ChatGPT-User, Perplexity-User and Claude-User. Blocking them means the assistant tells that person it couldn't open your page.

Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train future models. Letting them in or not is a business decision, and blocking them doesn't affect whether you get cited today. We cover both sides of that decision in our post on blocking AI crawlers.

Our AI crawler directory lists each bot, who runs it and which job it does. For the rest of this guide, "let AI crawlers through" means the first two groups, plus the third only if you've chosen to allow training.

Step 1: Work out which layer is answering

A request to your site can pass through several layers before it reaches the code that builds your page. Any of them can say no.

1. A network in front of your site, such as Cloudflare. It sits between the whole internet and your server, and it can block before anything else sees the request.
2. A cloud firewall that belongs to your hosting platform, such as AWS WAF or the Vercel Firewall.
3. A firewall run by your web host. Many shared and managed hosts run one at the server level, with settings you may not be able to see.
4. A security plugin inside your site, such as Wordfence on WordPress. This is the last layer, running inside your site's own software.

You may have more than one. A site on Vercel with Cloudflare in front of it has two separate firewalls, each with its own bot settings.

To see which services sit in front of your site, look at the response headers your homepage sends back. In Chrome, open the developer tools, go to the Network tab, reload the page, click the first request and read the "Response Headers" list. A few telltale signs:

• server: cloudflare or a cf-ray header means Cloudflare is in front.
• An x-vercel-id header means Vercel is serving the site.
• An x-amz-cf-id header means the site goes through Amazon CloudFront, which is often paired with AWS WAF.

A WordPress security plugin won't show up in headers. Check your plugin list. If you can't tell what your host runs at the server level, ask their support directly: "Do you run a firewall or bot blocker in front of my site, and can you show me what it blocked?"

Step 2: Find the block in the right record

Each service keeps its own record of what it refused. That record is where you find out whether the real crawlers are being turned away, and by which setting. Your server's own log, the one our guide to reading server logs walks through, only shows requests that got past every layer in front of it. A crawler stopped at Cloudflare never appears there at all, which is exactly why a quiet log can be misleading.

Here's where each service keeps its record, and what each one can block. Work through the ones you found in step 1.

Cloudflare

Cloudflare's record is Security Events. In the dashboard, open your domain, go to the Analytics page and choose the Events tab. It's available on every plan and keeps up to the last 31 days, though the free plan shows a sample of events rather than all of them. Add a filter on the user agent containing SearchBot, PerplexityBot or -User, and each event shows what action was taken and which Cloudflare feature took it. That last part is what you need, because Cloudflare has five different places that can stop an AI crawler, and the fix depends on which one did.

1. The AI bot policy

This is the setting built specifically for AI crawlers, under Security Settings. Since September 2026 it sorts AI bots into three groups (Search, Agent and Training) and lets you block or allow each one. We explain that change, and the catch that blocking Training now also catches Googlebot and Bingbot, in our post on website builders and AI crawlers. The short version: set Search to allow, set Agent to allow, and only block Training if you've decided to and understand what else it catches.

2. Bot Fight Mode

This is the free plan's bot protection, and it causes more confusion than any other setting, because it doesn't play by the same rules as the rest of Cloudflare. Cloudflare's documentation for Bot Fight Mode says plainly: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules."

In other words, there is no exception list. If Bot Fight Mode is challenging a crawler you want, Cloudflare's own guidance gives two options: turn Bot Fight Mode off, or move to a paid plan and use Super Bot Fight Mode instead, which does allow exceptions. You'll find it under Security Settings, filtered by "Bot traffic."

If your Security Events show Bot Fight Mode acting on AI crawlers, decide whether the scraping protection is worth more to you than being readable by AI search. For most small business sites, it isn't.

3. Super Bot Fight Mode (Pro plan and up)

The paid version has separate settings for "definitely automated" traffic, "likely automated" traffic, and verified bots, the crawlers Cloudflare has confirmed are who they say they are. Make sure verified bots are allowed. Unlike Bot Fight Mode, Super Bot Fight Mode can be skipped by a custom rule, which brings us to the next place.

4. Your own custom rules and rate limits

Custom firewall rules are the most common cause of an AI block nobody remembers making. A rule written years ago to block "all bots", or all traffic from cloud hosting networks, will catch AI crawlers today, because they run on exactly those networks.

The clean fix is an exception rule that runs first. Create a custom rule with this expression:

(cf.client.bot)

and set its action to Skip, skipping the remaining custom rules and Super Bot Fight Mode. cf.client.bot is Cloudflare's field for "this request comes from a known good bot or crawler", which means Cloudflare has checked the network address, not just the name. Then move the rule to the top of the list.

If you'd rather let through only AI bots, Cloudflare also has a cf.verified_bot_category field with values such as "AI Assistant" and "AI Search". Treat these with some caution: Cloudflare now keeps them mainly for older rules, and its newer Search, Agent and Training groups can't be used in custom rules at all. For most sites, the simple known-bot exception above is safer.

Rate limiting rules deserve a separate look. A crawler reads many pages quickly, which is exactly what a rate limit is designed to stop. Cloudflare's own rate limiting documentation warns that applying these rules to verified bots can hurt your search visibility. Every plan can use the Verified Bot field in a rate limiting rule, so add and not cf.client.bot to the expression of any rule that might catch crawlers.

5. AI Crawl Control

AI Crawl Control (it used to be called AI Audit) is Cloudflare's dashboard for AI crawler activity, available on all plans. It shows which AI services visit, how often, and whether they're following your robots.txt, and it lets you set allow or block rules per crawler. It's the quickest place to see at a glance whether OAI-SearchBot and PerplexityBot are reaching you, and the most likely place to find a per-crawler block somebody set and forgot.

AWS WAF

If your site runs on Amazon's cloud behind AWS WAF, the setting to check is the Bot Control rule group, and it has a surprise in it.

Bot Control sorts bots into categories (search engines, link checkers, monitoring services and so on). For almost every category, it only blocks bots it can't verify. A verified Googlebot passes the search engine rule untouched.

The AI category works differently. AWS's Bot Control documentation says the CategoryAI rule "applies the action to all matches, regardless of whether the bots are verified or unverified." Its default action is Block. So if you turned on Bot Control and left it at its defaults, every AI crawler is blocked: the training ones, but also OAI-SearchBot, PerplexityBot and the user-triggered fetchers, genuine or not.

To find out whether it's happening, look at the sampled requests for your protection pack (web ACL) in the AWS console, or your WAF logs if you've turned them on, and look for the CategoryAI rule name or the label awswaf:managed:aws:bot-control:bot:category:ai.

To fix it:

1. In the Bot Control rule group settings, change the CategoryAI rule's action from Block to Count. AWS calls this a rule action override. The rule still labels AI traffic, it just stops blocking it.
2. If you want to keep blocking training crawlers, add your own rule after the Bot Control group that blocks requests carrying both the AI category label and the name label of the specific training bot you want to keep out. AWS adds a bot:name: label to each bot it recognises, so you can match on the exact crawler.

Doing it this way keeps the protection AWS gives you against fakes. A scraper pretending to be PerplexityBot won't carry the verified label, so the rest of Bot Control still treats it as an unverified bot.

Vercel

Vercel's firewall has two settings that touch AI crawlers, both under Firewall, then Rules, in your project dashboard. Both are off unless someone turned them on.

The AI Bots ruleset is one switch for every AI crawler. Vercel's managed rulesets documentation says that in Deny mode it "blocks all traffic identified as coming from AI bots", and Vercel's bot documentation makes clear that covers bots crawling for training, for search, and for fetches a person asked for. There is no setting inside it for search only. If it's on Deny, ChatGPT search, Perplexity and Claude search are all locked out along with the training crawlers. Switch it to Log if you want to see the traffic without blocking it.

If you need to block training crawlers but allow search ones, leave the ruleset off and write your own custom rule instead, blocking the specific training crawlers by user agent. Vercel runs your custom rules before its managed rulesets, so a custom rule takes priority.

Bot Protection challenges visitors that don't behave like a browser, which crawlers never do. Vercel says it automatically leaves out verified bots, Google's crawler among them, so the known AI crawlers in Vercel's verified directory should pass. If you find one being challenged anyway, a custom rule with a Bypass action lets it skip the check.

One extra catch: Vercel says Bot Protection doesn't work properly when another network such as Cloudflare sits in front of your Vercel site. If that's your setup, you have two bot systems judging the same visit, and you need to check both records.

WordPress security plugins

On WordPress, the security plugin is often the only firewall the site has, and its defaults weren't written with AI crawlers in mind. Wordfence is the most widely installed, so it's the example here, but the same questions apply to any security plugin with bot or rate limiting settings.

Wordfence's rate limiting sets how many pages a visitor or crawler may load per minute before it's cut off. Its rate limiting documentation has a dedicated option for Google because "Google crawlers are special", letting verified Google crawlers through without limits. There is no equivalent setting for OpenAI, Anthropic or Perplexity. Their crawlers count as ordinary crawlers and fall under the general crawler limits.

So the two settings to check are "If a crawler's page views exceed" and "If a crawler's pages not found (404s) exceed." Wordfence itself flags limits of 60 page views per minute or lower as "Very strict" and likely to block friendly bots. If yours are set that low, raise them.

To see what's actually being blocked, open Wordfence's Live Traffic view and filter for blocked requests. If AI crawler names keep appearing there, your limits are catching them.

Other plugins use different names for the same idea: "bad bot" lists, "block fake crawlers", user agent blocklists. Look for any list that names AI crawlers, and any setting that treats anything which isn't Google or Bing as suspicious.

Akamai, Imperva, DataDome and other enterprise bot managers

If your company uses one of the large bot management services, the settings live in a private customer portal, and the public documentation doesn't spell out the steps the way Cloudflare and AWS do. All three recognise AI crawlers. DataDome keeps a public directory of AI crawlers, for example, and Akamai says its known-bot list includes OpenAI's crawler and dozens of other AI crawlers. But how each one is treated is set per account, often by a security team that has never been asked about AI search.

So the job here is a conversation, not a setting. Ask whoever manages it these four questions:

1. Which AI crawlers are we currently blocking or challenging? Ask for the list by name, not by category.
2. Do we treat search crawlers and training crawlers differently? If the answer is "we block AI bots", it almost certainly means both.
3. How do we confirm a crawler is genuine? You want the answer to involve the operator's published network addresses or a signed request, not just the name.
4. Can you show me the last month of requests from OAI-SearchBot, PerplexityBot and Claude-SearchBot, and what we sent back? This is the same record check as every other service, just done by someone else.

Make the exception the right way

Whichever service you use, there's a wrong way to open the door: allowing any visitor whose user agent says GPTBot or PerplexityBot. That lets every scraper wearing those names straight through, which is exactly what your firewall was there to stop.

Allow verified crawlers instead. Every service in this guide has a way to do it: Cloudflare's known-bot field, AWS's verified label, Vercel's verified directory. Where you're writing the rule yourself, match the name and the network address. OpenAI, Anthropic and Perplexity each publish their crawlers' addresses as a file your firewall or developer can read; our server logs guide lists where each one lives.

Perplexity's crawler documentation says that behind a firewall "you may need to explicitly whitelist Perplexity's bots", and recommends matching both the user agent and the published address ranges. It also says to always use the current lists, because they change.

That's also why a hand-copied list of addresses is a trap: lists change, and a rule built from last year's copy quietly starts refusing the real crawler. Wherever you can, use the service's own verified-bot feature, which the vendor keeps up to date for you.

Step 3: Confirm it worked

After you change a setting, the crawlers have to come back before you can see the result, and they don't visit every site every day. Give it at least a week, then go back to the record you used in step 2.

What you want to see is the search crawlers and user fetchers turning up over several days, and your firewall's record showing them allowed rather than challenged or blocked. Your server log, if you can get one, is the second confirmation that they're receiving real pages.

If they still never appear, the block is somewhere you haven't looked yet. Go back to step 1 and check for a second layer. Also re-read your robots.txt: our guide to robots.txt rules for AI crawlers covers the file-level mistakes, which are easy to miss once you've decided the firewall is the problem.

This is the very first gate in our AI readiness maturity model. Until the crawlers can open your pages, nothing else you do for AI search, from schema to content, gets read at all.

Common mistakes

Testing with a fake user agent and trusting the result. As covered above, your request isn't the real crawler, and a good firewall knows that. A pass proves nothing; a fail proves nothing. Only the firewall's record of the real visit counts.

Fixing one layer and stopping. A site behind Cloudflare and running Wordfence has two places that can block. Opening one doesn't open the other.

Blocking "AI bots" as one group. Most one-switch AI blocks, including Vercel's AI Bots ruleset and AWS's CategoryAI rule, catch search crawlers and user fetchers along with the training crawlers. If you only meant to stop training, use a setting or rule that names the training crawlers.

Forgetting rate limits. A crawler that's allowed in but cut off after a few pages is nearly as invisible as one that's blocked. Check rate limits as well as block rules.

Assuming free plans have nothing switched on. Cloudflare's free Bot Fight Mode is one click to enable and has no exception list. If someone turned it on to stop a spam problem two years ago, it's still running.

FAQ

Does my firewall read my robots.txt?

No. Your firewall decides before the request reaches your site, and it works from its own settings, not your robots.txt file. A crawler blocked by the firewall gets refused at the door even though your file says it's welcome. The only way to see this is in the firewall's own record of blocked requests.

How do I know if Cloudflare is blocking ChatGPT?

Open your domain's Security Events (Analytics, then the Events tab) and filter by user agent for OAI-SearchBot and ChatGPT-User. Each event shows which Cloudflare feature acted on the request. Check the AI Crawl Control page too, which shows AI crawler activity per bot.

Should I turn off Bot Fight Mode?

If it's challenging AI crawlers you want, and you're on the free plan, turning it off is the only fix, because Cloudflare doesn't allow exceptions to it. On a paid plan, switch to Super Bot Fight Mode, which allows verified bots and can be skipped by a custom rule.

Will letting AI crawlers through make my site less secure?

Not if you allow verified crawlers rather than names. Verification checks that the visit really comes from OpenAI, Anthropic, Perplexity or Google, so scrapers using those names are still caught by the rest of your firewall.

Does allowing GPTBot mean my content gets used for AI training?

Allowing GPTBot does let OpenAI use your pages for training. If you don't want that, keep GPTBot blocked and allow OAI-SearchBot and ChatGPT-User, which handle ChatGPT search and user requests. The same split works for Anthropic and Perplexity. Blocking a training crawler doesn't stop you being cited.

Do I need a paid plan to let AI crawlers through?

Usually not. On Cloudflare's free plan you can turn off Bot Fight Mode, write custom rules and use AI Crawl Control, which is on every plan. Exceptions inside Super Bot Fight Mode need the Pro plan or higher. Vercel's AI Bots and Bot Protection rulesets are available on all plans. AWS Bot Control is itself a paid rule group, so if you have it, you're already paying for it.

How long after fixing it will AI crawlers come back?

There's no fixed time. Crawlers revisit on their own schedules, and a site that has been refusing them for a while may be visited less often to begin with. Check your firewall's record after a week, and give it a few weeks before deciding the fix didn't work.

Is there a tool that checks this for me?

Not completely, because no outside tool can send a request as the real crawler. Tools can check your robots.txt, your page structure and whether your pages load at all, and that rules out most other problems quickly. The firewall question is answered by your firewall's own record.

Rule out everything else first

A firewall block is only one of several reasons an AI engine might not be reading your site, and it's the one that takes the longest to confirm. Our free AI readiness check rules out the rest in one pass: what your robots.txt says to each AI crawler, whether your pages load and can be read without JavaScript, and whether AI can tell who runs your business. If all of that comes back clean and AI search still isn't finding you, your firewall's record is the next place to look.