Can AI Crawlers Actually Reach Your Site? The Discover Pillar Explained

Before an AI system can understand your business, cite your content, or answer a question about you accurately, it first has to physically get to your page. That sounds too obvious to write down, until you realize how many sites fail this step without anyone noticing, because the site looks completely fine to a human visitor and completely different to a crawler. Discover is the first of the four AI Readiness pillars, and it answers one question: can AI crawlers reach this site at all?

It's worth 15 of the 100 total points in an AI Readiness score, smaller than Understand's 45, but that's not because it matters less. It's because Discover problems, once found, are usually the fastest of the four pillars to fix. The hard part isn't the fix. It's noticing the problem exists, since almost nothing about a Discover failure is visible from a normal browser.

What Discover Actually Checks

Discover looks at four things, roughly in the order a crawler would encounter them:

Whether your sitemap exists and is reachable. Not just present, actually discoverable. A crawler typically looks for it in one of a few places: a sitemap.xml or sitemap_index.xml at your domain root, a Sitemap: line inside robots.txt pointing to it, or, failing both, a fallback crawl starting from your homepage and following links. Each of those is progressively less reliable than the last, so a missing sitemap doesn't necessarily mean total invisibility, but it does mean a crawler is working harder and covering less than it should.

What your robots.txt file actually says. This is the single most common self-inflicted Discover problem. A robots.txt rule written broadly, a wildcard Disallow meant to block a staging environment, or a blanket rule left over from a site migration, can quietly block AI crawlers along with whatever it was originally meant to block, and nobody notices because the rule was never revisited after the situation that prompted it went away.

The HTTP status codes your pages actually return. A page that redirects through two or three hops before landing on its final URL, or one that intermittently returns a server error, is harder for a crawler to reliably index, and AI search crawlers, which often index in near-real-time to answer live queries, tend to have less patience for that than a traditional search crawler doing a slower, more thorough pass.

Whether your content depends on JavaScript to exist at all. This is the one that catches modern, well-designed sites off guard, and it deserves its own section.

The JavaScript Problem: Why AI Might Be Reading a Blank Page

Here's the failure mode that makes Discover worth checking even on a site that looks fast and modern: a page can render beautifully in a browser and be almost entirely empty to a crawler that doesn't execute JavaScript the way a browser does.

This happens because a lot of modern site architecture, plenty of React and Vue-based builds, some headless CMS setups, some page builders, ships a nearly blank HTML shell and then fills it in with JavaScript after the page loads. A browser runs that JavaScript automatically, so a human visitor never notices anything is different. A crawler that fetches the raw HTML and doesn't run the JavaScript sees the shell and nothing else: no headings, no body text, no schema markup, nothing that tells it what the page is about.

Not every AI crawler behaves the same way here, and that inconsistency is exactly the problem, you can't assume "it probably renders fine" and be right for every system that might visit. The practical fix isn't necessarily abandoning your framework; it's server-side rendering or static generation for the content that actually needs to be discoverable, which is a well-trodden path for headless WordPress setups specifically because this exact problem is common there.

You can get a rough first read on this yourself: view your page's source (not the rendered DOM, the actual response your server sends) and see how much of your real content is actually present versus how much only shows up after JavaScript runs. If your headings, body copy, and schema are missing from that raw source, a JavaScript-blind crawler is seeing what you're seeing minus everything that matters.

The AI Crawlers Worth Knowing About

For Discover purposes, what matters isn't memorizing every bot name, it's understanding that AI crawlers fall into three different jobs, and blocking one has very different consequences than blocking another. Training crawlers collect content for a future model and generally crawl in bulk without urgency. Search and retrieval crawlers build the live index a real AI answer gets assembled from, blocking one of these has an immediate, direct effect on whether your content can be cited by that system today. User-triggered fetchers are invoked only when a real person explicitly hands your URL to their assistant.

The practical takeaway for Discover: check your robots.txt against this three-way distinction, and make sure a broad "block everything" rule left over from an old staging setup or an incident response isn't catching search or retrieval bots you never meant to exclude. Platform-specific bot names and exact configuration details for ChatGPT, Gemini, and Perplexity are covered in depth in the existing per-platform citation guides, so this post won't re-walk that ground; it's focused specifically on the Discover-pillar question of reachability, not platform-by-platform citation tactics.

One general note worth keeping regardless of platform: user-agent strings can be spoofed, so if you ever need to verify a visitor is genuinely who it claims to be, a reverse DNS check or a published IP range is more reliable than trusting the header alone, not usually necessary for a small business, but worth knowing if something looks off in your logs.

The Crawler-vs-Browser Comparison

One of the more useful diagnostics for Discover is a direct comparison: fetch your own page as a plain, non-JavaScript request, the way a crawler would, and compare it against what a full browser renders. This catches two distinct problems at once.

The first is the JavaScript-dependency issue already covered above. The second is subtler and, when it happens, usually unintentional: some server configurations, CDNs, or bot-detection rules end up serving a different version of a page to a request that looks like a bot than to one that looks like a browser. Sometimes this is deliberate (aggressive bot-blocking that doesn't distinguish between malicious scrapers and legitimate AI crawlers); sometimes it's an accidental side effect of a security or performance tool configured too broadly. Either way, the result is the same from the crawler's side: it's not seeing your real page.

This is worth checking specifically because it's invisible from a normal browsing session, you have to actually fetch the page the way a crawler would to notice anything is wrong.

Running this comparison yourself doesn't require special tooling for a first pass. A command-line request tool that doesn't execute JavaScript, or even a browser extension that lets you view a page as a plain text fetch, will show you the raw response. Compare the headings, body text, and any schema markup you'd expect to see against what actually comes back. If a page that should clearly describe your services returns something closer to a loading skeleton, that's a genuine Discover-pillar failure, not a cosmetic issue, and it's one worth fixing before investing further effort in the content itself, since none of that content is doing any work for AI systems that can't see it in the first place.

Sitemap Discovery: What "Reachable" Actually Means

A sitemap existing somewhere on your server isn't the same as a sitemap being discoverable. Crawlers generally try a few methods, in a rough order of reliability, and it's worth understanding all of them rather than assuming the first one covers you.

The most reliable path is a Sitemap: line inside robots.txt, pointing at your actual sitemap URL. This is the method most crawlers check first, because it doesn't require guessing a filename or location, it's stated explicitly, in the one file almost every crawler already fetches as a matter of course.

The second path is a standard filename at your domain root, sitemap.xml or sitemap_index.xml being the most common conventions. A crawler that doesn't find a Sitemap: line in robots.txt will often try these locations directly, but this depends on the crawler actually checking, which isn't guaranteed the way an explicit robots.txt declaration is.

The least reliable path is a homepage link-crawl fallback: if neither of the above produces a sitemap, some crawlers will start at your homepage and follow internal links outward, building an implicit map of your site as they go. This works, but it's slower, it's easy for pages several clicks deep to be missed entirely, and it depends heavily on your internal linking actually connecting everything that matters, an orphaned page with no internal links pointing to it may never be found this way at all.

The practical implication: don't rely on the fallback. Declare your sitemap explicitly in robots.txt, keep it current as you add and remove pages, and don't assume that because a page is technically live, it's necessarily going to be found.

It's also worth checking what your sitemap actually contains, not just whether it exists. A sitemap that was generated once and never regenerated will list old URLs that no longer exist and omit new ones that do, which doesn't just create broken links for a crawler to hit, it also means your newest, often most important content might not be discoverable through this path at all.

What This Looks Like When It's Working Correctly

A site with a healthy Discover pillar has a fairly boring, unremarkable technical profile: a sitemap declared in robots.txt and kept current, no accidental blanket-blocking rules, clean HTTP status codes with no unnecessary redirect chains, and content that exists in the raw HTML response rather than depending entirely on client-side JavaScript to appear.

None of that is exciting to write about, and that's rather the point, Discover isn't a pillar where you're trying to do something clever or differentiated. It's the pillar where you're trying to make sure nothing accidental is standing in the way. The businesses that fail it usually aren't trying to hide from AI crawlers; they inherited a rule from an old project, built on a framework that wasn't originally designed with crawler visibility in mind, or simply never checked.

Assuming "it looks fine to me" means it's fine. The entire Discover pillar exists because a page can look completely normal in every browser you'd ever test it in and still be functionally invisible to a crawler. Visual inspection tells you nothing about this.

Leaving a broad robots.txt rule in place after the reason for it is gone. A rule written to keep a staging site out of search results, or to block an aggressive scraper during an incident, often just stays there indefinitely, quietly blocking AI crawlers nobody meant to exclude.

Treating every AI bot the same way. Blocking a training crawler and blocking a search/retrieval crawler have very different consequences. Making that decision without knowing which category a bot falls into means you might accidentally cut yourself out of live AI answers while trying to make a much narrower, more defensible choice about training data.

Never actually checking the raw response your server sends. Most Discover problems are only visible if you look at the page the way a crawler does, the raw HTML, not the rendered page in your browser's inspector after JavaScript has already run.

How to Check and Fix Discover Issues on Your Own Site

• Confirm your sitemap exists and is reachable at a standard location, and that it actually lists your real, current pages rather than a stale export from months ago.
• Read your robots.txt file directly, not from memory of what you think it says, and check it against the AI crawlers relevant to your business, training bots, search/retrieval bots, and user-triggered fetchers are each worth a deliberate decision, not a single blanket rule.
• View your page's raw source (not the browser-rendered DOM) for your most important pages and confirm your real content, headings, body text, schema, is actually present in that raw response, not just added afterward by JavaScript.
• Watch your server logs for AI crawler user-agents to see who's actually visiting and how often, which tells you more about your real-world Discover status than any single point-in-time check.
• Re-check after any migration, redesign, or CDN/security configuration change, these are exactly the events that tend to introduce Discover problems that weren't there before.

This is also exactly what a proper AI readiness audit checks first, since nothing else in your readiness score matters if a crawler can't reach the page at all. AI Schema Gen's audit tool runs the raw-vs-rendered comparison automatically and flags robots.txt issues specifically, rather than leaving you to check each one by hand.