llms.txt Doesn’t Work: 97% of Files Get Zero Requests

Ahrefs pulled server logs for 137,210 domains. In May 2026, 97% of the llms.txt files those sites publish got zero requests. Not from AI search, not from anything else (Ahrefs, 15 June 2026). So if you are weighing llms.txt for an AEO or GEO program, skip it. The file will not change whether ChatGPT, Perplexity, or Google’s AI surfaces quote you. Instead, robots.txt decides that, and most teams have never read theirs.

That answer stings, because llms.txt has become the first move on every “optimize for AI search” checklist. It ships in an afternoon. The work looks technical. Better still, it hands you an artifact to show in a status meeting. But none of that is the same as working.

The llms.txt numbers leave almost no room to argue

Ahrefs checked every domain in its Web Analytics tool with traffic in May 2026. That came to 137,210 sites. Then it looked at each root for an llms.txt returning HTTP 200. It also made sure the file was real Markdown, not a soft 404. Finally, it counted every request to that path.

About 28% of those domains ran a file, or roughly 38,360 sites. Of those, only about 1,100 saw any traffic at all. So 97% sat untouched for a full month. Ahrefs notes that its users skew technical, so it treats 28% as an upper bound. In other words, the gap is probably wider across the rest of the web.

The bots that read llms.txt are not the ones you wanted

Of the requests that did land, 96% came from bots. Notably, the mix is what should end this debate. Here is the split across all requests to valid files:

• SEO audit tools: 21.7%
• Unidentified bots: 14.9%
• General web crawlers such as Googlebot: 13.1%
• Tech profiling tools: 11.6%
• AI agents and agentic infrastructure: 10.5%
• GEO/AEO tracking tools: 5.8%
• AI training crawlers: 5.3%
• AI retrieval bots (OAI-SearchBot, PerplexityBot, Claude’s search fetcher): 1.1%, or 233 requests in total

Read that list again. The biggest reader of llms.txt is software built to tell you whether you have an llms.txt. Roughly a fifth of the traffic is the AEO industry auditing itself. Meanwhile, the bots that build the answers users see made one request in ninety.

Ahrefs adds a detail that reframes the file. Claude-Code, Anthropic’s coding agent, out-fetched every AI retrieval bot, every AI assistant, and every training crawler. So the file does get read. It just does not get read for search.

The 404 control group is the part nobody quotes

The best evidence in the study is the test almost every write-up skipped. Ahrefs also logged hits to llms.txt paths that returned 404, on sites with no file at all. Valid files drew 96% bot traffic. Missing files drew 98% human traffic. Furthermore, the AI bot share of those 404s was zero.

That is a control group, so it settles the question. If AI systems hunted for this file, they would throw 404s on the sites that lack one. They throw none. Clearly, no AI system goes looking. Rather, the file turns up by chance, found by crawlers that were already on the page.

Two more studies landed on the same answer

One dataset is a finding. Three make a pattern.

EZY.AI put the file on 83 sites, then logged crawlers for twelve weeks, from 27 April to 19 July 2026. In that window, OpenAI fetched robots.txt 3,990 times and llms.txt 7 times. Anthropic: 3,120 against 9. Perplexity hit robots.txt 775 times, yet never touched llms.txt once. Meta was the outlier at 193 fetches. Google managed 67. EZY’s verdict is the one worth keeping: “robots.txt is where AI access is actually decided today.”

Otterly.AI ran a separate 90-day log study on a site with a correct file. Across 62,100+ AI bot visits, /llms.txt drew 84. That is about a tenth of a percent. It is also three times worse than a normal content page there, which drew around 265. Otterly found no lift in AI bot activity at all.

Google has said as much for over a year. Here is John Mueller on Bluesky in June 2025: “FWIW no AI system currently uses llms.txt.” Then, a year later on Reddit, he had not budged: “it’s purely speculative for now (the file has existed for years, yet none of the AI systems use it).”

llms.txt keeps spreading on no evidence at all

Even so, the file keeps spreading. Originality.ai tracks more than three million sites. It counted 4,088 llms.txt files in June 2025, then 36,120 in May 2026. That is 8.8x growth in twelve months. Besides that, the variants grew faster off smaller bases. llms-full.txt went from 23 sites to 2,463, while ai.txt went from 4 to 397. Rankability sampled the Tranco top 1,000 in June 2026 and found 8.7% running one.

This is what a tactic looks like when it spreads on hunch rather than proof. It resembles a standard people already trust. It costs an afternoon. Then nobody checks the logs. In fact, the 5.8% of requests from GEO/AEO tools is the tell. An industry is grading homework that none of the graders can show matters.

There is one case where shipping it still makes sense

Do you publish developer docs? Is your product used by engineers running coding agents? Then ship it. Note that this is not a search play. Rather, it is a developer-experience play, and the Ahrefs data backs it directly. Coding agents, led by Claude-Code, are the heaviest real readers of the file. A clean map of your docs saves those agents tokens. It also gets your API described right inside someone’s editor.

For everyone else the file is a rounding error. Worse, you then have to keep it in sync with your site forever. The cost is not the twenty minutes of writing. Instead, the cost is the stale file two years from now, pointing agents at dead URLs. Add the attention it pulled from work that compounds.

Audit robots.txt this week instead

Spend that hour on the file AI crawlers demonstrably fetch thousands of times:

1. Read your robots.txt as an AI access policy, because that is now its job. Most were written before AI crawlers existed.
2. Split retrieval bots from training crawlers. GPTBot and ClaudeBot gather training data. Meanwhile, OAI-SearchBot and PerplexityBot fetch pages to answer live queries. So a blanket “block AI” rule can quietly cut you out of answers.
3. Check that the crawlers you allow get HTTP 200. A generous robots.txt means nothing when your WAF returns 403 to the same agents. Therefore, read your edge logs, not your config file.
4. Put the answer in the first 100 words of any page aimed at a question. Retrieval systems pull passages. So a page that buries its answer never gets quoted.
5. Publish something only you can publish. Your own numbers, benchmarks, and client data. Original data is the one asset that makes a page worth citing rather than paraphrasing.

None of that yields a satisfying screenshot. Still, all of it changes whether a machine can reach, parse, and quote you. That is the whole job.

Want help with what your AI visibility really rests on?

Karma Group audits the technical and editorial layer behind AI search. We start with your crawler access rules, then the pages you want quoted. So if you would rather know than guess, get in touch.