Should You Block AI Crawlers From Your Site? The Trade-offs

You did not agree to this. AI companies now worth hundreds of billions of dollars built their models partly on the open web, your site included, without asking and without paying. When you find out you can push back with a few lines in a text file, the urge to do it is completely understandable.

Whether it is the right move is a separate question, and the honest answer for most sites is "probably not, and here is why it matters less than it feels like it should." This post is the strategic version of that decision. The exact tokens and rules, if you decide you do want to block, are in our guide to allowing and blocking AI crawlers.

The one distinction that decides everything

AI crawlers do two jobs that matter here, and blocking each one has a completely different cost.

Retrieval crawlers build the live index an engine answers from. Block one and you are not in that engine's answers at all, while competitors who did not block it still are. That is a direct, measurable loss.

Training crawlers collect pages to train a future model. Block every one of them and nothing changes about whether you show up in an AI answer today. There is no visibility cost.

So "should I block AI crawlers" is really two questions. For retrieval crawlers the answer is almost always no, because being absent from AI answers is a real cost with no offsetting benefit for most businesses. For training crawlers the answer is "it does not hurt, so the only question is whether it helps." And mostly, it does not help much either.

What blocking training crawlers would actually protect

Your content is probably already in the data. Common Crawl has published a free, open archive of the web every month since 2011, and it is one of the most common starting points for training a large language model. If your site is a few years old, versions of your pages are already in that corpus, already downloaded, already used. A block you add now does not reach back.

The models answering questions today were trained months or years ago. Your block applies to the next training run, not the one that already happened.

The crawlers that would ignore your block are the ones you would most want to stop. robots.txt is a polite request. The first-party crawlers from OpenAI, Anthropic, Perplexity, and Google honour it. Bulk scrapers frequently do not, and Cloudflare, which sees a large slice of web traffic, reports that training makes up nearly 80% of AI crawler activity and that much of it is "aggressive at times, often ignoring directives found in robots.txt files."

Real enforcement lives a layer up, at your CDN or firewall, where a crawler can be blocked by verified network address and behaviour rather than by the name it gives. Some hosts have gone further: Cloudflare now blocks AI training crawlers by default on new domains and runs a pay-per-crawl system that lets a site charge for access instead of choosing between free and blocked. None of that is configured in robots.txt, and all of it is more work than most sites should take on unless the content is genuinely worth defending.

What a robots.txt block on training crawlers does do: stop future collection by the companies that respect the rule, cut crawl load from those same well-behaved bots, and put a clear, dated statement on record that you did not consent. Whether those are worth anything depends entirely on what kind of site you run.

Here is the same thing as a table, since the three options get muddled together in most advice:

The middle row is where most of the debate should sit, because it is the only one that changes a real variable without costing you anything visible.

The case for blocking

Your content is the product. Original research, journalism behind a paywall, a licensed dataset, technical docs you charge for, creative work you sell. If people pay for your content, letting a model ingest it to paraphrase for free is giving the product away. This is the strongest case, and it is why most large news and research publishers now block the training crawlers.

Crawl volume is a real cost for you specifically. Training crawlers request far more pages than they send visitors. Cloudflare has measured the ratio of pages crawled to visitors referred at hundreds to one for some AI companies and many thousands to one for others. On a large or frequently updated site that is bandwidth and server load you absorb for nothing. If your own logs show that pattern, blocking the polite crawlers and rate-limiting the rest is a straightforward operations call.

You have a licensing strategy. Big publishers block first, then sell access. The New York Times signed an AI content deal with Amazon reported at up to $25 million a year while still suing OpenAI over past training use. Reddit, News Corp, and others have their own deals. The catch is that these exist because the content is worth licensing at scale. A block is a precondition for that conversation, not a way to start one, and most sites will never be positioned to have it.

Consent, as a principle. You did not agree, you do not want to take part, and that is a legitimate stance even when the practical effect is small. Go in knowing the effect is small.

There is a fair counterargument here worth acknowledging. Any one site blocking changes almost nothing, but that is true of every individual site, and if enough of them acted together the picture would look different. Collective pressure is how the licensing deals happened at all. If you block on principle, that is the argument for it: not that your single robots.txt file matters, but that a norm is worth contributing to. Whether that outweighs your own visibility is a values call, not a technical one, and this post is not going to make it for you.

The case against blocking

For a marketing site, blog, local business, SaaS product, or store, blocking is usually the wrong call, and not only because of the retrieval-crawler cost.

You want to be in AI answers. More product research, comparisons, and "who should I hire for X" questions start inside an assistant now. If your category is discussed there and you are not eligible to be cited, that is ground your competitors hold.

"Block everything" almost always catches the retrieval crawlers. The blunt rules people copy from forum threads routinely sweep in OAI-SearchBot and PerplexityBot next to the training bots. Most reachability failures that keep sites out of AI search are an over-broad block, not a deliberate one.

Blocking does not make AI go quiet about you. Ask an assistant about a business whose crawlers are blocked and it still answers, from directories, cached pages, a competitor's "alternatives to" post, old training data. You have not left the conversation, you have removed your own voice from it and left the AI describing you from worse sources. For a business that is usually worse than being described from your own current pages.

There is no payment either way for a small site. The trade is not "get paid or get scraped." It is "invisible and untrained" versus "visible and trained." Framed plainly, most businesses would take the second.

Allowing has a cost too, and it is worth naming. AI answers often summarise your content without sending a click, so you can be cited, paraphrased, and used as a source while your traffic from that query drops. This is a genuine downside of allowing, and it is the one people are usually reacting to. It is still not a reason to block, because blocking does not recover the click, it just removes you as the source while the summary gets written from someone else. The response that works is structuring your pages so the summary is built from your facts rather than guessed and your brand is the one named in it, not withdrawing from the index.

Your competitors' choice is part of yours

This decision is not made in isolation. If you block the retrieval crawlers and everyone else in your category allows them, you are the one company an assistant cannot describe from current information when a buyer asks it to compare options. The buyer does not see a gap where you should be. They see a shorter list that does not include you.

Look at who actually blocks. Originality.ai found over a third of the world's top 1,000 sites blocking GPTBot by mid-2024, up from under 12% when it launched a year earlier, and the blocking is heavily concentrated among news publishers: a Reuters Institute study found nearly half of top news sites block at least one AI crawler. Marketing sites, SaaS companies, local businesses, and stores mostly do not, because for them the visibility is worth more than the training opt-out. If your site looks more like the second group than the first, that pattern is a signal.

What being blocked looks like from the AI's side

Say a prospect asks an assistant "is [your company] any good for X" and you have blocked its retrieval crawler. It does not say it cannot help. It builds an answer from a two-year-old cached homepage, a review-site profile with your old pricing, a competitor's comparison post, a forum thread. The prospect reads a confident summary of your business assembled from everyone except you.

Allowing the crawler does not guarantee a flattering answer, but it puts your current positioning, real pricing, and own description of what you do into the pool the answer draws from. The fix for a business is more of your accurate content reaching the model, not less.

Blocking is reversible, but not on your schedule

A block is not a one-way door. You can lift it whenever you change your mind. What you cannot control is the timing of the recovery. When you unblock a retrieval crawler it re-crawls on its own cycle, which for a modest site can run to weeks, and the engine's answers only reflect your content once that crawl and its reprocessing finish. A block you hold for two months is not a two-month absence, it is two months plus however long the crawler takes to come back and catch up. If you are only blocking to make a point, weigh that lag against how much the point is worth.

Google is a single switch

Google has no partial option. AI Overviews and AI Mode are part of Google Search, built from the same Googlebot index as the regular results, and no token separates them. Google-Extended only governs Gemini model training, not Search, so it does not pull you out of AI Overviews. Blocking Googlebot removes you from everything. If reducing AI Overview usage is the actual goal, snippet-length controls are the only lever, and our guide to appearing in AI Overviews covers what moves it.

Measure before you decide

Most of this argument turns on facts specific to your site, so check them before choosing:

• Read your access logs for the AI crawler names over a few weeks. If training crawlers are pulling thousands of pages a day, the crawl-cost case is real for you. If they barely appear, it is not.
• Check what you are already doing. Open your live robots.txt and your CDN's bot settings. A surprising number of sites are already blocking crawlers they never meant to, and never knew because nobody looked.
• Look at whether your category shows up in AI answers. Ask a few real buyer questions in ChatGPT and Perplexity. If competitors are being cited and you are not, that tells you what allowing is worth before you consider blocking.
• Check whether you already have leverage. If AI engines cite you often and your content is genuinely distinctive, you may be in a position to license rather than just block. If they rarely mention you, a block is a statement with no one on the other end of it.

A decision framework

Marketing site, blog, local business, SaaS, or store: allow the retrieval crawlers and user fetchers. Allow or block the training crawlers by preference, knowing it barely moves anything. Put the effort into being clearly described instead.

Block the training crawlers if your content is something people pay for, or your logs show AI crawl volume that costs you real money. The first is a decision about protecting a product; the second is about load.

Only block a retrieval crawler if you can say the specific reason out loud and you accept being absent from that engine's answers while competitors are present.

If you do block, do it properly. robots.txt for the crawlers that honour it, network-level rules for the ones that do not. The companion how-to has the current tokens and the exact rules.

Before you decide, see what AI crawlers are actually doing on your site. Run a free AI readiness check for a robots.txt audit against the current crawler list, plus a look at what a bot really sees when it visits.