Before any content strategy, check whether these systems can reach your site at all. A surprising number of teams working hard on AI visibility have a Disallow line quietly making the effort pointless, and many of them added it deliberately, without realising what it did.
This is the practical guide: which crawlers matter, what to allow and block, where robots.txt stops working, and what actually earns a citation once access is sorted. If you want the conceptual grounding first, see what AEO actually means.
Three families of AI crawler
Nearly every mistake in this area comes from treating "AI bots" as one thing. They're three, with very different implications.
Training crawlers collect data to build model weights. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider, Meta-ExternalAgent, Amazonbot. They crawl in bulk without urgency. Blocking them keeps your content out of training corpora.
Retrieval crawlers build the indexes AI engines cite from at query time. OAI-SearchBot (ChatGPT Search), PerplexityBot, Claude-SearchBot, DuckAssistBot. These are the ones that determine whether you can appear in a live AI answer.
Conversational agents fetch a page when a user explicitly asks an assistant to visit it. ChatGPT-User, Claude-User, Perplexity-User.
Then two opt-out tokens sit slightly apart: Google-Extended and Applebot-Extended. These cover Gemini and Apple Intelligence training respectively, and disallowing them doesn't affect your search visibility. Blocking Google-Extended does not remove you from AI Overviews; AI Overviews are served through normal Google Search crawling.
The mistake most sites make
Blocking retrieval and agent crawlers is what actually removes you from live AI answers. Blocking training crawlers is a content-rights decision with essentially zero citation cost.
Most sites that "block AI bots" with one blunt rule end up blocking the wrong class, cutting themselves out of citations while achieving little of what they intended.
It's often not even deliberate. Sites built or last audited before 2023 frequently block AI crawlers by default through aggressive Cloudflare, Sucuri, or CDN configurations that treat OAI-SearchBot and PerplexityBot the same as malicious scrapers. Nobody made a decision; a security default made it for them.
The asymmetry is worth stating plainly: if you accidentally block a retrieval bot, your brand disappears from the AI answers that crawler powers. If you accidentally allow a training crawler you meant to block, your content ends up in a training corpus. Both matter, but only one is invisible until someone asks why you never appear in ChatGPT.
The genuinely useful consequence: because training and retrieval are separate agents, you can opt out of being used for training while remaining eligible for citation. Block GPTBot, allow OAI-SearchBot. That nuance is available to you and most sites don't use it.
A working robots.txt
Here's a selective configuration: allow retrieval and agent bots, make a deliberate choice on training.
# Retrieval crawlers: allow these to stay eligible for AI citations User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: DuckAssistBot Allow: / # Conversational agents: user-initiated fetches User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Perplexity-User Allow: / # Traditional search User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # Training crawlers: a content-rights decision, no citation cost either way User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / # Generative training opt-outs: do not affect search visibility User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / Sitemap: https://example.com/sitemap.xml
The training block is genuinely optional and reasonable people disagree. Allowing training crawlers means your content and positioning inform what models know about your category. Blocking them is a defensible content-rights position. What isn't defensible is making the choice by accident.
If you'd rather allow training too, change those Disallow: / lines to Allow: /. The retrieval section is the part that shouldn't change.
Check your CDN and WAF as well
Your robots.txt can say one thing while your infrastructure does another, and the infrastructure wins.
Cloudflare, Sucuri, and similar services apply bot management independently of robots.txt. Rules written to stop scrapers routinely catch legitimate AI retrieval crawlers, because from a WAF's perspective they look similar. Verify that what your robots.txt intends is what your edge actually permits.
Two practical checks: search your access logs for the declared user agents, and confirm you're seeing requests from the bots you've allowed. Absence of OAI-SearchBot or PerplexityBot in logs, when your robots.txt permits them, points at an edge rule.
A grep pattern covering the main declared agents:
GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot| PerplexityBot|Perplexity-User|Google-Extended|Amazonbot|CCBot|Applebot| meta-externalagent|DuckAssistBot|bingbot
Where robots.txt stops working
Three honest limits, because treating the file as a guarantee leads to bad decisions.
Compliance is voluntary. A directive only works if the crawler reads and honours it. The major documented bots (Googlebot, GPTBot, ClaudeBot, PerplexityBot) publish their user agents and respect standard directives. Others don't. Bytespider in particular has been widely documented ignoring robots.txt, and one analysis found it accounted for the overwhelming majority of AI crawler traffic across a large customer base, much of it disregarding disallow rules. For non-compliant bots the only real defence is at the server or WAF level.
User agents can be spoofed. Anyone can send a header claiming to be PerplexityBot. If you're making access decisions that matter, verify with a reverse DNS lookup rather than trusting the string.
User-directed fetches are a grey zone. When someone asks ChatGPT or Perplexity to summarize a specific URL, the tool fetches that page on the user's behalf. Providers often treat this as user-directed access rather than crawling, so robots.txt rules may not apply the way site owners expect.
The llms.txt question
You'll see llms.txt recommended frequently. The honest assessment: it is not an access control mechanism, and the vendors operating the crawlers have either said nothing about it or declined to support it. Sites publishing llms.txt and llms-full.txt generally report no material effect on AI retrieval.
It costs little to publish and may become meaningful if adoption grows. But don't treat it as a substitute for robots.txt configuration, and don't let anyone sell it to you as one.
Access is necessary, not sufficient
Once crawlers can reach you, the question becomes what earns a citation. This is less certain territory, and we'd rather be honest about the confidence levels than sell you a formula.
Be the clearest explanation, not just the source. Answer engines may cite a page that isn't the original source if that page explains the answer more clearly. A manufacturer's spec sheet can lose to a third-party comparison that presents the same facts in a cleaner, more quotable form. If you're a primary source, authority alone doesn't earn the citation; you have to explain your own facts at least as well as the people writing about you.
Lead with the answer. State it plainly near the top, then add depth below. Content that buries the response is harder to extract accurately.
Use evidence. Verifiable statistics, named sources, direct quotes, observed results. The most-cited academic work on generative engine optimization found that citing sources, adding statistics, and including quotations improved visibility in generative responses. Secondary coverage often flattens this to "up to 40%," which is worth treating as directional rather than a promised return.
Publish non-commodity content. Google's own guidance contrasts generic listicles with specific, first-hand accounts. If forty pages make the same point, a synthesis system doesn't need to cite any of them. If one page holds information available nowhere else, citing it becomes necessary.
Keep pages fast and redirect chains short. Retrieval crawlers operating at query time have less tolerance for slow responses and multi-hop redirects than a patient training crawler does. Treat query-time retrieval as having a tighter performance budget than traditional crawling.
Structure for extraction. Headings matching how people ask, distinct claims in distinct blocks, self-contained explanations that survive being lifted out of context.
Platform differences worth knowing
The four major surfaces behave differently enough that a single mental model will mislead you.
Google AI Overviews and AI Mode. Served through normal Google Search infrastructure, so ordinary Googlebot access governs eligibility; there's no separate AI crawler to allow. Google states no additional requirements or special optimizations apply. Practically, this means AI Overview visibility rests on the same foundations as organic visibility: indexability, quality, and relevance. The one AI-specific control worth checking is your snippet directives, since restrictive nosnippet or max-snippet settings can limit how a page is featured in AI experiences.
ChatGPT Search. Powered by OAI-SearchBot's index, with ChatGPT-User fetching pages when someone asks about a specific URL. GPTBot is separate and training-only. The important implication: allowing GPTBot does nothing for ChatGPT Search citations, and blocking it costs you nothing there. If ChatGPT visibility is the goal, OAI-SearchBot is the agent that matters.
Perplexity. PerplexityBot builds the index; Perplexity-User handles direct fetches. Perplexity is unusually citation-forward: it displays sources prominently, which makes it the easiest surface on which to observe whether your work is landing. Worth using as your canary: if you're cited nowhere, checking Perplexity first is the fastest diagnostic.
Microsoft Copilot. Runs on Bing's index, so Bingbot access is the gate. Microsoft has been the most forthcoming vendor about structured data's role here: Fabrice Canel, Principal Product Manager at Microsoft Bing, confirmed in March 2025 that structured data helps Microsoft's large language models understand content. If you want one surface where markup arguably earns its keep for AI purposes, this is the documented one.
The practical takeaway: Google needs no special access work beyond good SEO hygiene, while ChatGPT and Perplexity need explicit retrieval-crawler permission that many sites have never granted.
Presence beyond your own site
This is the part most guides skip, and it may matter more than anything on your own domain.
AI answers about a category are frequently assembled from third-party sources (comparison articles, review platforms, community discussions, industry roundups) rather than from vendor websites. If someone asks an answer engine to recommend a tool in your category, the systems answering are often working from what other people wrote about you.
That has a practical consequence: being genuinely present in the places your category gets discussed matters alongside optimizing your own pages. Not manufactured mentions, which are transparent and increasingly filtered, but real participation, real coverage, real inclusion in the roundups and comparisons people actually read.
It also means the misrepresentation risk is real. If outdated information about your product circulates on third-party sites, that's what gets summarized. Monitoring what answer engines say about you is a maintenance task, not a one-off audit.
A 30-minute audit
If you do nothing else, do this. It's the highest return-per-minute work available in this area, because access failures are both common and completely fixable.
1. Read your robots.txt. Open yourdomain.com/robots.txt and search for: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Claude-User, CCBot, Google-Extended, Applebot-Extended, Bytespider. Any Disallow: / under a retrieval or agent bot is removing you from that engine's answers. Note what you find before changing anything.
2. Check for a blanket block. A User-agent: * block with restrictive rules, or a catch-all AI block added at some point by a plugin or a well-meaning developer, is the most common cause. Plenty of CMS SEO plugins shipped "block AI bots" toggles that make no distinction between training and retrieval.
3. Grep your access logs. Use the user-agent pattern above and confirm you're actually seeing hits from the retrieval bots you permit. Permitted in robots.txt but absent from logs means something upstream is blocking them.
4. Check your CDN and WAF rules. Cloudflare bot management, Sucuri, and similar tools apply policy independently of robots.txt. Look specifically for bot-category rules that would catch AI crawlers alongside scrapers.
5. Test a page as an agent would see it. Ask ChatGPT or Perplexity to summarize one of your key URLs. If it can't retrieve the page, you have a live demonstration of the problem.
6. Check snippet directives. Look for nosnippet, max-snippet, and data-nosnippet in your templates. Restrictive settings can limit how pages appear in AI experiences, and they're frequently inherited from a template rather than chosen.
7. Decide training policy deliberately. Having seen what's currently blocked, make an actual decision about training crawlers rather than leaving whatever happens to be there.
Most sites find at least one problem in this list, and the fix is usually a few lines of configuration.
Measuring it
Server logs tell you a crawler visited. They don't tell you whether you were cited. This is the fundamental gap. Log analysis confirms access is working (genuinely useful, since that's the failure mode you can actually fix), but it says nothing about whether you appeared in an answer.
Google Search Console includes AI Mode and AI Overview traffic in Performance reports, within the "Web" search type rather than as a separate category. Clicks and impressions from AI surfaces are folded into your existing totals, so there's no AI-only breakout to isolate.
Track citations manually. Query the major answer engines periodically for your priority topics and record whether you appear, and whether what's said about you is accurate. Unglamorous and the most reliable signal available. Third-party citation tracking tools exist and can automate this; evaluate their methodology before trusting their numbers.
Watch branded search and direct traffic for downstream effects when you're cited without a click.
A note on the statistics circulating in this area: you'll see confident figures like "sites that unblocked AI crawlers saw a 186% traffic lift in 90 days." Treat these the way you'd treat any vendor-published number with no methodology attached. The underlying advice, allow retrieval crawlers, is sound for reasons that don't need an invented percentage.
Where structured data fits
Briefly, since we sell schema tooling and have written about this at length in does schema markup help with AI Overviews.
Google states no special schema.org markup is required for AI Overviews or AI Mode. Microsoft has said structured data helps its models understand content for Copilot. Structured data makes you eligible for rich results in the types Google still supports, removes ambiguity about what your content means, and asserts entity facts through sameAs and stable @id values.
It's useful infrastructure, worth checking against our pricing for what's included on the free plan. It is not the lever that gets you cited, and crawler access matters considerably more. If you fix one thing after reading this, fix the robots.txt.
Frequently Asked Questions
AI Schema Gen generates and validates structured data across 827+ schema types: the comprehension and entity layer beneath your AI visibility work. Start free, read the documentation, or browse the schema types directory.
Generate perfect schema in 30 seconds
AI Schema Gen handles everything automatically, free to start.
Get Started Free