llms.txt: 97% of Files Get Zero AI Requests

The finding says your site has no /llms.txt. Most tools that raise it will tell you to go write one, and most articles about it open by explaining how urgent this is for AI search visibility.

Here is the evidence instead. In May 2026, Ahrefs looked at server logs across 137,000 domains and found that 97 percent of llms.txt files received zero requests. Not few. Zero. And among the small remainder that were fetched at all, the single largest category of requester was SEO audit tools, at 21.7 percent of requests. Actual AI retrieval bots accounted for 1.1 percent.

Read that until it lands. The main thing reading llms.txt files today is the software that checks whether you have an llms.txt file.

Why this matters for AI traffic

It mostly does not, and Google has stopped being coy about it. From Google’s own guidance on AI features: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” The same page says there are “no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary,” and spells out what eligibility actually takes: “a page must be indexed and eligible to be shown in Google Search with a snippet.”

That is the whole mechanism. Be crawlable, be indexed, be worth quoting. A file at your root does not add a lane.

The proposal itself is reasonable and was never a ranking play. Jeremy Howard published it in September 2024 to solve a real engineering problem: “Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety.” An llms.txt is a curated markdown index, an H1, a summary, and lists of links, meant to hand a model the good version of your site instead of your navigation and cookie banner. That is a sensible idea. It is just not a thing search engines promised to consume, and two years on, the logs say they mostly do not.

What llms.txt is actually good for

One real consumer exists, and it is not a search engine. Coding agents fetch documentation at inference time: Cursor, Copilot, and Claude pulling your docs while someone builds against your API. For that audience an llms.txt is genuinely useful infrastructure, because it says which pages matter and hands over clean markdown instead of rendered HTML. Anthropic ships one for its own docs for exactly this reason, aimed at IDE agents and MCP integrations rather than at search.

So the question is not “do I want AI visibility”. Everyone does. The question is narrower: does anyone build against my site? If you publish developer documentation, llms.txt is real plumbing with a real user. If you sell dog grooming in Tulsa, it is a file nobody will request, and the audit that flagged it is echoing a fashion rather than a finding.

When is it worth writing, and when should you skip it?

Write one when:

You publish developer docs, an API reference, or anything a coding agent gets pointed at. This is the case the spec was designed for, and it works.
You already generate markdown, so the file is close to free and stays current on its own.

Skip it when:

You are a normal business site hoping to show up in AI Overviews. Google has told you in plain language that this file is not the path, and the logs agree.
Maintaining it by hand would compete with work on the pages themselves. A stale llms.txt is worse than none: it is a curated index that lies, aimed at nobody.

The honest tradeoff: this is close to free if your site already emits markdown, and close to worthless for search visibility either way. I would ship one for a docs site, skip it for a marketing site, and spend the hour on being crawlable and worth quoting instead, because that is what Google says the requirement actually is.

Get AI Traffic does not write this file for you, and that is deliberate rather than an oversight: an agent-generated index of a site nobody’s agent will fetch is the definition of busywork. If you decide you want one, it is a static file at your root, and the two related findings (an incomplete llms.txt, or a missing llms-full.txt expansion) are only worth acting on once you have decided the first one is.

Frequently asked questions

Does llms.txt help me rank in AI Overviews? No. Google’s guidance is explicit that you don’t need to create machine readable files, AI text files, or markup to appear in AI features, and that eligibility comes from being indexed and snippet-eligible. Nothing about a root file changes that.

Do OpenAI or Anthropic read it? Not for search or citation. Both point site owners at robots.txt for crawler management. Anthropic publishes one for its own documentation, aimed at coding agents. In the Ahrefs log study, GPTBot accounted for 4.51 percent of llms.txt requests and ClaudeBot for 0.80 percent, against 21.7 percent from SEO audit tools.

Then why does the audit flag it? Because the file is cheap to check for and the industry decided it was a best practice before anyone measured whether it was one. Treat this finding as a prompt to make a decision, not as a defect. For most sites the correct decision is to skip it.

What is llms-full.txt? An expanded variant that inlines the linked content rather than pointing at it, so a model gets the whole corpus in one fetch. Same logic applies: useful when a coding agent consumes your docs, pointless otherwise.

Is it going to matter later? Possibly. It costs little to add if your site already produces markdown, and if retrieval bots start honoring it the file is already there. That is a real argument for a docs site. It is not a reason to hand-maintain one for a site nobody builds against.

Nobody Is Reading Your llms.txt, Except the Tool That Told You to Write One

Why this matters for AI traffic

What llms.txt is actually good for

When is it worth writing, and when should you skip it?

Frequently asked questions

Sources