The Hidden noindex Tag That Blocks AI

When people worry about AI crawlers not reaching their site, they check robots.txt. It is the obvious file, it is easy to read, and a bad rule in it is easy to spot. So that is where the attention goes.

There is a second switch that does something quieter and more complete, and almost nobody checks it: the noindex rule. It does not sit in one file at your domain root. It hides in the page itself, or in a response header you never see in a browser, and it tells any crawler that respects it to fetch the page, read every word, and then keep none of it. A robots.txt block says "do not come in." A noindex says "come in, look around, now forget you were ever here."

For Google's AI answers, and increasingly for the AI search features inside ChatGPT and other assistants, a stray noindex is one of the most total ways to make a page vanish. This guide covers exactly what the rule does, the handful of accidental ways it gets switched on, and how to find and clear it on your own site.

noindex is not robots.txt, and the difference matters

These two controls feel similar and do almost opposite things.

robots.txt controls crawling. It is a single file at yoursite.com/robots.txt, and a crawler reads it before fetching anything else. A Disallow line means the crawler should not request those URLs at all. Because it never fetches the page, it never sees the content.

noindex controls keeping and showing. It lives on the individual page, either as a tag in the HTML head:

<meta name="robots" content="noindex">

or as an HTTP response header the server sends alongside the page, called X-Robots-Tag:

X-Robots-Tag: noindex

Either form says the same thing: you may crawl this, but do not store it in your index and do not show it in results. In Google's own words, "noindex is a rule set with either a <meta> tag or HTTP response header and is used to prevent indexing content by search engines that support the noindex rule."

Here is the catch that makes noindex so easy to miss. For a crawler to obey it, the crawler has to fetch the page first, because that is the only way to see the tag or the header. Google states it plainly: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler." So a noindex page looks completely healthy to every check that only reads robots.txt. The crawler is allowed in. The sitemap lists the URL. The page returns a clean 200 OK. And the page still never makes it into an AI answer, because the moment the crawler finished reading it, the tag told it to drop everything it just learned.

What a noindex actually does to AI systems

Google's AI Overviews and AI Mode. These are built on Google's main search index and crawled by Googlebot. They are not a separate system with their own opt-in. Google is direct about this in its AI features documentation: to limit what shows from your pages in Search, "use nosnippet, data-nosnippet, max-snippet, or noindex controls," and AI features are "integral to how Search functions." A page carrying noindex is not in Google's index, so it is not in normal search results, the "People also ask" questions, the AI Overview at the top of the page, or AI Mode. One tag removes it from all of them at once.

ChatGPT's search features. OpenAI's crawler that powers search inside ChatGPT is OAI-SearchBot, and OpenAI's publisher guidance says you can use "the noindex meta tag" to keep a page from being surfaced there, with the same condition Google has: "in order for the crawler to read a meta tag, it must be allowed to crawl the relevant page(s)." OpenAI also notes that if it already has your URL from another source and thinks the page answers a user's question, it may still show a bare link and page title, but not the content itself. The page stops being a source it can quote.

Other assistants. Perplexity, Claude's browsing, Gemini through Google Search, and the rest are less consistently documented, and you should not assume every AI crawler treats noindex identically. But the ones that behave like well-mannered search crawlers, which is most of the ones that matter for citations, treat it the way Google does: crawl, see the tag, do not keep the page.

There is one thing noindex does not necessarily stop, and it is worth knowing. AI model training is usually governed separately, by crawler-specific rules in robots.txt (Google-Extended, GPTBot, and so on), not by noindex. So a noindex is aimed squarely at the "can this page be found, quoted, and cited right now" question, which is exactly the question that decides whether AI answers mention you.

The Discover pillar of an AI readiness check asks whether a crawler can physically reach your page. A noindex sits one step past that: the crawler reaches the page fine, then is told the visit does not count. In a readiness score it lands in the Read pillar, and it is the heaviest single penalty there, because a page the crawler has been told to forget cannot do any work no matter how good its content or its schema is.

The accidental ways noindex gets switched on

Almost nobody adds noindex to a page they care about on purpose. It gets there by one of these routes, and every one of them is common.

The staging site checkbox that never got unticked. WordPress has a setting under Settings, then Reading, labelled "Discourage search engines from indexing this site." Developers tick it while a site is being built so the half-finished version does not show up in search. Since WordPress 5.3, ticking that box makes every page output <meta name='robots' content='noindex,nofollow' /> in the head. When the site launches, someone has to remember to untick it. Often nobody does, and the brand-new site goes live completely invisible to search engines and AI crawlers alike. The nofollow half makes it worse: it tells crawlers not to follow any link on the page either, so even the pages that are fine get discovered more slowly.

SEO plugin rules applied to a whole template. Yoast, Rank Math, and All in One SEO all let you set noindex on an entire class of pages at once: tag archives, author archives, paginated pages, a custom post type, media attachment pages, search results pages. Most of these defaults are reasonable. The problem is when the rule is wider than intended, or when a post type you actually want found ("Case Studies," "Locations," "Resources") was set to noindex once and forgotten. The plugin's own content analysis usually shows a green light on the page itself, because the page is fine. The template-level switch is set two menus away.

An X-Robots-Tag header added at the server or CDN. This is the nastiest version, because the rule is not in the HTML at all. It is added by the web server config (Nginx, Apache), by a hosting platform's header settings, by a CDN or edge worker (Cloudflare, Fastly, Vercel), or by a _headers file in the deploy. A rule meant to hide a /preview/ path or a non-production hostname gets written a little too broadly and starts matching live URLs. You can view the page source all day and never find it, because it only exists in the response headers. Our own audit crawler did not check for this until we added it, and when we did, we found sites scoring themselves as perfectly indexable while every page carried a header-level noindex.

Preview and non-production environments that share a codebase. A Next.js, Gatsby, or headless build often sets noindex when an environment variable says the deploy is a preview or a branch build. If production somehow inherits that variable, or a caching layer serves a preview-built page to real visitors, the tag ships to production with it.

"Private," "hidden," or "members only" page states. Many CMSs and membership plugins add noindex automatically to anything gated. That is correct for a real members area. It is a problem when a page was set to a restricted state during editing and never switched back, or when a "coming soon" or maintenance-mode plugin is left active on one section of the site.

The dead robots.txt line people still trust. You will see advice to put Noindex: lines inside robots.txt. Google stopped supporting that in September 2019 and it never worked with other crawlers. A Noindex: line in robots.txt today does nothing at all, which means anyone relying on it to keep a page out is also relying on it to let a page in, and it is doing neither. If you want a page indexed, this line is harmless. If you thought it was managing your indexing, it is not.

How to find a noindex on your own site

A noindex never changes how a page looks or loads, so clicking around the site tells you nothing. You have to look in the two places the rule can hide, and you have to check more than the homepage, because noindex is very often set per template.

Check the page source for the meta tag. Open the page, view its source (not the rendered inspector, the raw source), and search the text for robots. You are looking for any <meta name="robots" ...> or a bot-specific one like <meta name="googlebot" ...> that contains the word noindex. Do this on your homepage, one blog post, one product or service page, one category or tag archive, and one page two or three clicks deep. If different page types give different answers, you have a template-level rule.

Check the response headers for X-Robots-Tag. In your browser, open developer tools, go to the Network tab, reload the page, click the first request (the document itself), and read the Response Headers. Look for X-Robots-Tag. If you are comfortable with a terminal, curl -I https://yoursite.com/your-page prints the headers on their own. Again, test several page types and a couple of hostnames if you have a www and a non-www or a separate blog subdomain.

Read Google Search Console's Page Indexing report. This is the fastest way to see the scale of the problem. Open the Page Indexing report and look for the reasons "Excluded by 'noindex' tag" and "Page with redirect." Any URL you expected to be indexed that shows up under the noindex reason is a page currently removed from Google Search and its AI features. The URL Inspection tool, run on a single page, will also tell you "Indexing allowed? No: 'noindex' detected" and whether it saw the tag in the HTML or the HTTP header. The same Search Console workflow that surfaces schema problems surfaces this one.

Crawl the whole site with a tool that reports both. A site crawler (Screaming Frog, Sitebulb, or the crawl step inside an AI readiness audit) will fetch every page and report the meta robots value and the X-Robots-Tag value side by side for each URL. This is the only way to be sure a rule is not hiding on one obscure template out of forty.

Cross-check your sitemap. A page that is in your XML sitemap and also carries noindex is sending two opposite signals, and it makes a crawler trust the whole file a little less. If a URL should be found, it belongs in the sitemap without a noindex. If it genuinely should not be found, take it out of the sitemap too.

How to fix it, by where it lives

The fix depends entirely on what put the tag there.

WordPress "Discourage search engines" checkbox: Settings, then Reading, then untick "Discourage search engines from indexing this site" and save. The meta tag disappears from every page immediately.

Yoast: Search Appearance (or SEO, then Search Appearance), then the Content Types, Categories & Tags, and Archives tabs. For anything you want found, set "Show X in search results?" to Yes. Rank Math has the equivalent under Titles & Meta. All in One SEO has it under Search Appearance, then Content Types and Taxonomies.

A single page set to noindex by hand: every SEO plugin has a per-page advanced setting (Yoast calls it "Allow search engines to show this page in search results?"). Set it back to the default or to Yes.

An X-Robots-Tag header: you have to find the rule in whatever adds headers. Search your Nginx config for add_header X-Robots-Tag, your Apache config or .htaccess for Header set X-Robots-Tag, your next.config.js headers block, your _headers or vercel.json file, and your CDN or edge rules. Narrow the rule so it matches only the paths or hostnames it was meant for, or remove it.

A framework environment rule: find where the code decides to render noindex (usually a check on NODE_ENV, VERCEL_ENV, a NEXT_PUBLIC_ flag, or a "preview" boolean) and confirm production evaluates it to "index."

After the fix, use Search Console's URL Inspection tool to "Request Indexing" on a few key pages so Google recrawls them sooner. AI search crawlers do not have a request button, but they recrawl on their own schedule once the blocking signal is gone. There is no penalty to wait out and nothing to do beyond letting the recrawl happen.

Common mistakes

Fixing the meta tag and leaving the header, or the reverse. A page can carry noindex in both the HTML and the X-Robots-Tag header. Removing one and not the other leaves the page just as blocked. Always confirm both are clear after a fix.

Adding a robots.txt Disallow to "reinforce" a noindex. This is the single most common way people accidentally lock a noindex in place forever. If you block the URL in robots.txt, the crawler can no longer fetch the page, so it can never see that you removed the noindex. The page stays out of the index indefinitely. When you want a page indexed, it must be crawlable and free of noindex at the same time.

Only checking the homepage. The homepage is almost never the one with the problem. Template-level rules hit archives, paginated pages, custom post types, and deep pages while leaving the front page clean. Check one of each.

Trusting the SEO plugin's page-level green light. The content analysis score on an individual post tells you nothing about a noindex set at the post-type or taxonomy level. Those are different screens.

Assuming a robots.txt audit covers it. A robots.txt checker, including many "is my site blocking AI" tools, cannot see a noindex tag or header by design, because those only exist on the page. A clean robots.txt report is not a clean bill of health here. Reviewing your robots.txt rules for AI crawlers and checking for noindex are two separate jobs.

Confusing noindex with nosnippet. They are different tools for different goals. noindex removes the page entirely. nosnippet and data-nosnippet keep the page in the index but stop Google from quoting text from it, which also removes your normal search snippet. If your goal is to stay in search but out of AI answers, that trade-off is the thing to understand, and noindex is not the lever for it.