Auditing Your XML Sitemap for AI Crawler Discovery

An XML sitemap is a plain list of the URLs on your site that you want a search engine to know about. It has been a quiet background chore for twenty years: a plugin generates one, you forget about it, and it usually works. The AI era has not changed how a sitemap works, but it has changed who is reading it and what a stale or messy one costs you. This is a guide to auditing yours, section by section, with the AI crawlers in mind.

Who actually reads your sitemap now

Start with an uncomfortable fact: the AI crawlers everyone worries about mostly do not document using your sitemap at all.

OpenAI's crawler documentation describes GPTBot, OAI-SearchBot, and ChatGPT-User purely in terms of robots.txt. It never mentions sitemaps. Perplexity's bot page and Anthropic's crawler page are the same: they tell you how to allow or block their crawlers, and they say nothing about how those crawlers find URLs in the first place. There is no published statement from any of the three that says "we fetch your sitemap.xml and crawl what is in it."

So where does the sitemap fit? Through the search engines the AI systems lean on.

• Google reads sitemaps, and Google's index feeds AI Overviews and AI Mode. A page Google has not discovered and indexed is not a candidate for a Google AI answer.
• Bing reads sitemaps, and Bing's index has powered a large share of ChatGPT's search results and other assistants' web lookups. A page Bing has not indexed is much harder to surface in those answers.
• The AI-native crawlers (GPTBot, PerplexityBot, ClaudeBot, and the rest) mostly discover pages the old-fashioned way: by following links, and by pulling from the search indexes above.

The practical version of this: your sitemap's job in the AI era is to make sure the two search engines that AI systems actually depend on have a complete, accurate list of your real pages. It is not a direct line to ChatGPT. It is an indirect but load-bearing one, through Google and Bing.

That also means a sitemap is not the whole discovery story. Whether a crawler can even reach the file, what your robots.txt says, and whether your pages return clean status codes all sit alongside it. We covered that wider picture in the Discover pillar guide; this post assumes a crawler can reach your site and focuses on the sitemap file itself.

Is your sitemap where crawlers look for it

A sitemap sitting on your server does nothing if nothing points to it. There are three ways a crawler finds it, in rough order of reliability:

1. A Sitemap: line in your robots.txt. This is the most reliable path, because robots.txt is the one file almost every crawler fetches first, and the line states the sitemap's location outright with no guessing. It should read Sitemap: https://www.example.com/sitemap.xml, with the full address, the right protocol, and the right subdomain.
2. A standard filename at your domain root. Most commonly sitemap.xml or sitemap_index.xml. Crawlers that do not find a robots.txt line will often try these, but "often" is not "always."
3. Direct submission in Google Search Console and Bing Webmaster Tools. This tells those two engines specifically, and it also gives you their error reports on the file, which is worth having.

Do all three. The robots.txt line covers every crawler that reads that file, and the Search Console and Bing submissions give you diagnostics you cannot get any other way.

One thing that used to be on this list and is not any more: the "ping" URLs. For years you could send an HTTP request to a Google or Bing endpoint to say "my sitemap changed, come look." Google shut its endpoint down in 2023, and requests to it now just return an error. Bing has moved the same job to IndexNow, covered further down. If you have an old plugin still pinging the dead endpoint, it is not causing harm, it is just doing nothing.

What belongs in the file, and what does not

This is where most sitemaps quietly rot. The rule is simple to state and easy to break: a sitemap should be one clean list of the canonical, indexable URLs that each return a 200 status, and nothing else.

Every entry that breaks that rule sends a mixed signal. You are telling the crawler "this URL matters, index it" while some other part of your site says the opposite. Do it on a handful of URLs and it is noise. Do it on hundreds and a crawler starts trusting the whole file less.

The usual offenders:

URLs that redirect. If /old-page now 301-redirects to /new-page, the sitemap should list /new-page, not /old-page. A sitemap full of redirects means a crawler spends its visit chasing hops instead of reading content.

URLs that return 404 or 410. Dead pages left in the file after they were removed. Common when the sitemap is a stale export rather than something regenerated on every change.

Pages with a noindex tag. If a page tells search engines not to index it, it should not also be in the sitemap asking to be indexed. Pick one. Usually the page is right and the sitemap entry is the mistake.

Non-canonical duplicates. Filtered category pages, paginated URLs, print versions, URLs with tracking parameters, the http version of an https page. If a page's canonical tag points somewhere else, only that somewhere-else URL belongs in the sitemap.

Staging or development URLs. A staging.example.com or example.com/dev/ URL that leaked into the production sitemap during a migration. Rare, embarrassing, worth checking for once.

Thin utility pages. Internal search-result pages, empty tag archives, login and cart pages. Nothing a crawler should be led to as priority content.

A tidy sitemap is not about perfection for its own sake. It is that every URL you leave in the file is a small vote for "this is real, current, and worth your time," and you want every one of those votes to hold up when the crawler checks.

Are the signals inside the file trustworthy

The XML sitemap format has four fields per URL: the address (loc), a last-modified date (lastmod), a change frequency (changefreq), and a priority (priority). Only two of them still do anything.

changefreq and priority are ignored. Google has said plainly that it does not use either, and the reasoning is obvious once you hear it: both are self-reported by the site owner, and site owners are not reliable judges of their own content's importance. Every page ends up marked "priority 1.0, changes daily." Bing treats them with the same skepticism. You can leave them in your sitemap or strip them out; either way they change nothing.

lastmod matters, but only if it is honest. Google uses the last-modified date in a sitemap when it is "consistently and verifiably accurate," which it checks by comparing the claimed date against what it finds when it actually fetches the page. If your dates hold up, Google leans on them to decide what to re-crawl. If your site has a track record of lastmod values that do not match reality, Google stops trusting the field across your whole domain.

The way sites break this is almost always automated: a sitemap plugin sets every URL's lastmod to the moment the sitemap was generated, so the file claims the entire site changed today, every day. That is the fastest way to get the field ignored. An honest older date is more useful than a fake fresh one, and no lastmod at all is better than a dishonest one.

Because dates are their own rabbit hole, and because they show up in three places on every page, not just the sitemap, we gave them a full guide of their own. The short version for sitemap purposes: lastmod should move when the page's main content genuinely changes, and not when you edit a tag, bulk-run a plugin, or roll the copyright year.

Sitemap index files and the size limits

A single sitemap file is capped at 50,000 URLs or 50MB uncompressed, whichever comes first. Most sites never come close. If you do, you split the list across several sitemap files and add a sitemap index file, which is a sitemap of sitemaps: one small XML file that lists the location of each individual sitemap, and the URL you then declare in robots.txt and submit to Search Console.

Even well under the limit, a site with clearly separate sections benefits from splitting the sitemap by type: one for posts, one for pages, one for products, one for the document library. Two reasons. It makes Search Console's coverage reporting readable, because you can see at a glance that "products" has 40 URLs indexed out of 900 while "posts" is fine. And it lets a crawler that only wants your freshest content go straight to the sitemap that holds it.

Most SEO and CMS plugins produce an index file automatically once you pass a threshold. The thing to check is that the index file itself is the URL declared in robots.txt, not one of the child sitemaps, and that every child listed in the index actually loads.

Telling Bing, and everything downstream of Bing

Google gets most of the attention, but for AI visibility Bing is doing quiet heavy lifting, because its index has fed ChatGPT's search results and other assistants' web lookups. Getting your pages into Bing's index is a real part of AI discoverability, not an afterthought.

Two moves cover it:

Submit your sitemap in Bing Webmaster Tools, the same way you do in Search Console. You can import your Search Console setup directly if you would rather not reverify.

Use IndexNow for new and changed pages. IndexNow is an open protocol where you send a small notification the moment a page is published or meaningfully updated, and the page gets picked up for crawling much faster than waiting for the next scheduled sitemap check. Bing, Yandex, Naver, Seznam, and Yep all consume it; submitting to the shared endpoint distributes to all of them at once. Google does not participate, so IndexNow supplements your sitemap and Search Console rather than replacing them. Many SEO plugins have IndexNow support built in and just need switching on.

Does anything on your site actually depend on the sitemap

For a small, well-linked site, the honest answer is "not much." If every page is reachable in a click or two from your homepage, and your navigation and internal links are solid, crawlers will find your content with or without a sitemap. Google's own documentation calls a sitemap "merely a hint" and is explicit that submitting one "doesn't guarantee" the URLs get crawled or indexed.

The sitemap earns its keep in specific situations:

• New content, where you want it discovered in hours rather than whenever a crawler next wanders past a link to it.
• Large sites, where link-following alone will miss pages that sit many clicks deep.
• Orphan pages, ones with few or no internal links pointing at them. A landing page you only share by email, an old but still-valuable article nothing links to any more. The sitemap may be the only way a crawler learns these exist.
• Sites with a weak internal linking structure, where the sitemap partly compensates for navigation that does not connect everything.

If you find pages that only exist in your sitemap and nowhere in your internal links, treat that as a linking problem to fix, not a reason to lean harder on the sitemap. A page worth indexing is a page worth linking to, and internal links do more for how a crawler weighs a page than a sitemap entry ever will.

Common mistakes

Assuming the plugin has it handled. Most sitemap plugins produce a reasonable file by default, but "reasonable" drifts. A plugin update changes what post types are included, a setting gets toggled, a migration leaves a second plugin also generating a sitemap. Open the actual file once or twice a year and read it.

Two plugins, two sitemaps. Running Yoast and another SEO or sitemap plugin at once, both generating a sitemap at different URLs, with different contents. Pick one, turn the other's sitemap off, and make sure robots.txt points at the survivor.

Leaving noindex pages in the sitemap. The single most common mixed signal. If a page should not be indexed, it should not be in the file asking to be.

Stamping every URL with today's date. Covered above. It gets your lastmod field discounted site-wide, which is worse than having no dates at all.

Submitting once and never looking at the report. Search Console and Bing Webmaster Tools both tell you how many sitemap URLs they actually indexed, and flag the ones they rejected and why. That report is the point of submitting. Check it after any big content change.

Building a separate "AI sitemap." There is no such standard. AI crawlers do not consume a special sitemap format, and the file that helps them is the same XML sitemap that helps Google and Bing. If you want to give AI systems extra guidance, that is what llms.txt is proposed for, and its actual value is a separate debate, not a sitemap question.

How to audit your sitemap

• Find it. Check https://yourdomain.com/robots.txt for a Sitemap: line, and load the URL it points at. Confirm the file loads, is actual XML, and uses the right protocol and subdomain.
• Read a sample of the URLs. Open ten or fifteen entries. Every one should load with a 200 status, be the canonical version of its page, and not carry a noindex tag. If you hit redirects, 404s, or parameter URLs, the file needs regenerating.
• Check the count against reality. Roughly how many real, indexable pages does your site have? If the sitemap lists far more, it is probably including archives, tags, or duplicates. Far fewer, and a post type is being left out.
• Look at the lastmod dates. Pick a few pages you know you have not touched in months. If their lastmod says this week, your sitemap is faking freshness and the field is working against you.
• Open the Search Console and Bing sitemap reports. Note the "discovered" versus "indexed" gap per sitemap. A section with a large gap is where to dig.
• Turn on IndexNow if your platform supports it, so new and updated pages reach Bing and its downstream consumers quickly.
• Fix orphan pages by linking to them, not by relying on the sitemap to carry them alone.

A clean sitemap is one part of whether AI systems can reach your site at all, alongside your robots.txt, your rendering, and your crawler allow-list. If you want a single check across the whole set, run a free AI readiness audit and it will flag sitemap, crawlability, and structure issues together.