llms.txt vs. Schema Markup: Which One Do AI Engines Actually Use?

Two files keep coming up in every conversation about getting found by AI. One is llms.txt, a plain text file you put at the root of your site to tell AI systems what your site is about. The other is schema markup, the structured data you embed inside your pages to describe what's on them. Plenty of guides present these as two paths to the same destination, or worse, as a choice: add the file, or add the markup, and either way the robots will start quoting you.

That framing is wrong, and it's costing people real time.

These two things do different jobs, get read by different systems, and have wildly different amounts of evidence behind them. One of them has been measured across 137,000 domains and turned out to be almost entirely unread. The other is read every day by systems you already care about, but not for the reason most people think. Neither one is a switch that makes ChatGPT recommend you.

Here's what each file actually is, what the measurements say, and how to decide which one deserves your afternoon.

The Short Answer

AI search crawlers almost never fetch llms.txt. In a study of 137,210 domains published by Ahrefs in June 2026, 97% of the llms.txt files that existed received zero requests during May 2026. Not zero requests from AI. Zero requests from anything at all.

Schema markup does get read, but not by the AI chat interface directly. It's read by the search infrastructure sitting underneath a lot of AI answers, and Bing says outright that structured data is one of the clues it uses to understand a page. Google, meanwhile, says plainly that structured data is not required for its AI features.

Neither is a citation button. If you take one thing from this post, take that. We sell a schema markup product and we still won't tell you markup gets you cited, because Google's own documentation says it doesn't work that way, and we went through the evidence claim by claim separately.

The rest of this post is the reasoning, the numbers, and the one situation where llms.txt genuinely earns its place.

What llms.txt Actually Is

llms.txt was proposed by Jeremy Howard of Answer.AI and first published on 3 September 2024. The spec lives at llmstxt.org and was updated to a version 2 on 10 August 2026, so it is still an active proposal rather than an abandoned one.

The file itself is deliberately simple. It sits at your domain root, at /llms.txt, in the same spot you'd find robots.txt. It's written in Markdown, and the format asks for very little:

• An H1 with the name of the site or project. This is the only genuinely required part.
• An optional short summary in a blockquote.
• Any extra background you want to give.
• H2 sections holding a hand-picked list of links to the pages that matter, each with a short note about what it contains.

The stated purpose, in the spec's own words, is that "agents are expected to view or search llms.txt to find the information they need, then follow the relevant links." The idea is that instead of an AI agent crawling your whole site and burning through a pile of tokens on navigation menus and cookie banners, it reads one clean page that points it straight at the good stuff.

As an idea, it is genuinely sensible. The problem is not the idea. The problem is that the agents were never asked, and mostly, they don't do it.

Some sites also publish llms-full.txt, a much longer file containing the actual content rather than links to it. That's a related convention rather than part of the core spec, and everything below applies to it too.

What Schema Markup Actually Is

Schema markup is a description of your page written in a vocabulary machines already agree on, usually as a JSON-LD block inside the page's HTML. It says things like: this is an Organization, its name is this, its phone number is this, it has these social profiles, and it published this Article, written by this Person, on this date.

The vocabulary comes from schema.org, which was created in 2011 by Google, Microsoft and Yahoo together, and has grown to over 800 types covering nearly every kind of thing a website might describe.

There is one structural difference between the two files that explains almost everything else in this post, and it's worth stating on its own.

llms.txt is a claim you make about your site, stored somewhere else. Schema markup is a description attached to the thing it describes.

When you write llms.txt, you're writing a separate document that says "here is what my site is about." Nothing checks it against the site. You could describe a completely different business and the file would still be valid.

When you write schema markup, the description lives inside the page it describes, and it can be checked against what's visible there. If your markup says the price is $40 and the page says $60, that mismatch is detectable, and search engines treat it as a policy problem rather than a harmless typo.

John Mueller, then at Google, made exactly this point about llms.txt in a June 2025 comment:

AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag, this is what a site-owner claims their site is about.

The keywords meta tag comparison is pointed. That tag was abandoned by search engines decades ago for one reason: it was an unverified self-description, so it got abused into uselessness. Anything that is purely a claim, with no tie to the content it claims to describe, tends to end up in that same bin.

What the Measurements Say About llms.txt

This is the part most guides skip, because the numbers are not kind.

The Ahrefs study, June 2026. Ahrefs looked at 137,210 domains in its Web Analytics product that received traffic during May 2026. It checked each domain root for an llms.txt file returning a normal 200 response, then went through every request made to those /llms.txt paths and sorted them by user agent.

The headline finding: 97% of the llms.txt files that existed received zero traffic in May 2026.

Among the small remainder that did get requests, the picture doesn't improve much. Roughly 96% of those requests came from bots rather than people, and when the bots were identified, the biggest groups were not AI systems at all. SEO audit tools made up 21.7% of requests. Unidentified agents were 14.9%. General web crawlers were 13.1%. Tech profiling tools were 11.6%.

The AI crawlers people are actually hoping to reach were a thin slice. GPTBot accounted for 4.51% of requests, ClaudeBot for 0.8%, and OAI-SearchBot for 0.74%.

The Otterly experiment, 90 days. A separate test by Otterly.AI took a different approach: put a correctly built llms.txt on a site, then watch AI bot behaviour across the whole site for 90 continuous days.

Over that period the site received more than 62,100 AI bot visits in total. Of those, 84 went to /llms.txt. That's about 0.1% of all AI bot traffic to the site. For comparison, an average ordinary content page on the same site got around 265 AI bot visits in the same window, meaning the file built specifically for AI agents was visited roughly three times less often than a normal page that wasn't.

Their conclusion was that the presence of a correctly implemented /llms.txt file "did not correlate with any noticeable uptick in overall AI bot activity or shift in crawling patterns."

Two independent methods, same answer. One study counted requests across a very large sample of sites. The other watched one site closely for three months. Log analysis at scale and a controlled before-and-after both landed in the same place, which is about as much agreement as this field ever produces.

And the platforms themselves say so. Google's own AI optimization guide puts it in writing:

You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search.

OpenAI's own crawler documentation names its four bots, explains what each one does, and tells site owners what actually helps: allow OAI-SearchBot in your robots.txt and permit requests from the published IP ranges. There is no mention anywhere of reading a third-party site's llms.txt.

The Confusion That Keeps This Alive

If the evidence is this consistent, why does llms.txt keep getting recommended?

Mostly because of one genuinely confusing fact: the AI companies publish llms.txt files for their own documentation. OpenAI, Anthropic and Google all serve one for their developer docs. The spec site says so directly, and it's true.

This gets read as an endorsement. It isn't one, and the distinction matters more than it might seem.

Publishing a file is not the same as consuming one. OpenAI serving llms.txt at its docs site means OpenAI wants coding agents to read its documentation efficiently. It says nothing whatsoever about whether OpenAI's crawlers read your file when they visit your site. Those are two unrelated decisions made by two different teams for two different reasons.

It's the same shape as a company publishing an RSS feed. Doing so tells you they want their content syndicated. It tells you nothing about whether they subscribe to yours.

The second reason is quieter: a large share of llms.txt files were never a decision at all. Documentation platforms like Mintlify auto-generate llms.txt, llms-full.txt and related files for every site they host. So a lot of the adoption numbers people cite as momentum are really one platform's default setting, multiplied across its customers.

Where llms.txt Genuinely Works

Now the fair part, because there is a real use case in the data, and it's specific.

Look again at the Ahrefs breakdown of which AI agents actually fetched these files. Alongside GPTBot, one of the most active AI requesters was Claude Code, the coding agent. That is a meaningful detail.

Coding agents behave differently from search crawlers. A search crawler is out gathering the web broadly and has no particular reason to look for an unusual file at your root. A coding agent is pointed at a specific documentation site by a developer who wants an answer about a specific library, right now, and it benefits enormously from a compact index instead of a sprawling docs site full of navigation.

Mintlify, which hosts documentation for a large number of software companies, reports from its own analytics that nearly half of all traffic to documentation sites now comes from AI agents including Cursor, Claude Code, ChatGPT and Perplexity. On a docs site, llms.txt isn't a speculative bet on future crawler behaviour. It's serving a group of readers that already makes up a large share of the audience.

So the honest rule is this:

• If you run developer documentation, an API reference, or a technical knowledge base, publish llms.txt. Your readers include coding agents, those agents do fetch it, and the payoff is real today.
• If you run a marketing site, a local business site, an ecommerce store, a blog, or a service business, publishing llms.txt is close to a no-op. It will not hurt you, since systems that don't use a file simply ignore it. But the measurements say almost nothing will read it, so don't count it as work done toward AI visibility.

That's a narrower recommendation than "everyone should have one," and it's the one the evidence supports.

So What Do AI Engines Actually Read?

If llms.txt isn't the answer for most sites, it's fair to ask what is. Here's the honest version, kept short, because we've covered parts of it in depth elsewhere.

Your normal pages, fetched as HTML. This is the overwhelming majority of it. AI search crawlers request your pages the same way any crawler does. Which means the boring work matters most: your pages have to be reachable, they have to return clean status codes, and their content has to exist in the raw HTML rather than appearing only after JavaScript runs. That last one catches a surprising number of modern sites, and it's what crawler reachability comes down to.

Your robots.txt, which they do check. Unlike llms.txt, this file is genuinely read and respected, and it's where the highest-leverage decision actually sits. Blocking OAI-SearchBot or PerplexityBot removes you from those systems' live answers. That single line in a text file matters more than any amount of Markdown at your root.

Structured data, by the search infrastructure underneath. Bing states in its own documentation that "Bing works hard to understand the content of a page and one of the clues that Bing uses is structured data," and its Markup Validator handles schema.org, JSON-LD, Microdata, Microformats, Open Graph and RDFa. Since Microsoft's AI products sit on top of Bing's index, that's a direct line from your markup to an AI answer layer.

Google is more restrained, and its restraint deserves quoting accurately rather than spun:

Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add.

That's from Google's own AI optimization guide, and it's unambiguous. There is no AI-specific schema type, no separate AI index, and no markup that flags your content for special treatment in generated answers. Anyone selling you otherwise is guessing.

What structured data does earn you is different, and more concrete: eligibility for rich results in Google Search, comprehension signals for Bing and the systems built on it, and a machine-readable statement of who you are and how the things on your site relate to each other. That last one is the part we think is most underrated, and it's the difference between an AI reading a fact about your business and an AI guessing it from your prose and happening to get it right.

For a full accounting of what schema does and doesn't buy you in 2026, including the study that found no citation lift from adding JSON-LD, we worked through the cost and benefit properly.

Side by Side

Common Mistakes

Treating llms.txt as a substitute for crawler access. This is the expensive one. A site can have a beautifully written llms.txt and a robots.txt rule that blocks the exact crawlers deciding whether it appears in AI answers. The file at the root that AI systems definitely read is robots.txt. Get that right before writing anything else.

Reading vendor publication as vendor support. OpenAI and Anthropic publishing llms.txt for their docs is not a signal that their crawlers consume yours. Check what a company's crawler documentation says its bots do, not what its docs team ships.

Copying your sitemap into llms.txt. The point of the file was a short, curated, annotated list of the pages that matter most. Dumping every URL you have produces something no agent benefits from, at which point you've built a worse sitemap in a format fewer things read.

Letting llms.txt go stale. Because almost nothing fetches it, nothing tells you when it's wrong. A file listing pages you deleted last year is worse than no file, for the same reason a schema reference pointing at something that doesn't exist is worse than no reference: you've published a confident statement that happens to be false.

Believing anyone who promises citations from either one. Google's documentation is explicit that no markup and no AI text file is required to appear in its AI features. Any tool, agency or blog post attaching a specific percentage to schema and AI citations is quoting a number nobody has substantiated.

Adding markup that contradicts the visible page. Since schema markup is checkable against your content, a mismatch between what your JSON-LD says and what a visitor sees is a genuine risk rather than a harmless inconsistency. Prices, availability, ratings and dates all need to agree with the page.

Assuming your SEO plugin already settled this. Some do now ship llms.txt generation. Rank Math added support for it in 2026, for example, alongside AI traffic tracking. Having the feature switched on is not the same as it doing anything for a non-docs site.

How to Check and Fix

Work through these in order. They're deliberately sequenced by how much each one actually moves.

1. Read your robots.txt and confirm the retrieval crawlers are allowed. Fetch yoursite.com/robots.txt in a browser and look for anything blocking OAI-SearchBot, PerplexityBot, ChatGPT-User or Perplexity-User. These are the bots that decide whether you can appear in live AI answers, and a broad Disallow written years ago to hide a staging folder is the most common way sites block them by accident. This is a five minute check with a bigger payoff than everything else on this list combined.

2. View your raw page source and confirm your content is in it. Right click and choose "View Page Source," which shows the HTML your server actually sent, rather than opening developer tools, which shows the page after JavaScript has run. Your headings, your body copy and your JSON-LD should all be visible in that raw source. If they only appear in the rendered version, crawlers that don't run JavaScript are seeing an empty shell.

3. Decide honestly which kind of site you have. If it's developer documentation, write an llms.txt: an H1 with your product name, a one line summary, and H2 sections listing the pages that matter with a short note on each. Keep it curated. If it's anything else, skip it and spend the hour on step 4 instead.

4. Check whether your structured data actually says who you are. Most sites have some markup, usually whatever their SEO plugin emits by default, and most of it describes individual pages without ever establishing the business behind them or connecting the pieces together. Look for whether you have an Organization or LocalBusiness node at all, whether it carries a real name, logo, contact details and social profiles, and whether your content nodes reference it rather than floating unattached.

5. Make sure your markup matches your pages. Run a few important URLs through a validator and check the values against what's on screen. Pay closest attention to prices, stock status, ratings and dates, since those are the ones that change and go stale silently.

6. Re-check after any migration, redesign, or CDN change. These are the events that quietly introduce a blocking rule or break a rendering path. Nothing tells you when it happens.

If you'd rather not do steps 1, 2 and 4 by hand, that's precisely what our free audit runs: crawler access, the raw-versus-rendered comparison, and whether your entity data is present and connected, in a single pass.

Want to know what an AI system can actually work out about your business right now? Run a free AI readiness check and see your crawler access, rendering, and entity data in one report.