llms.txt: Does Your Business Need One?

Short answer: almost certainly not yet, and anyone charging you for one should have to show you the evidence.

We're an agency that does AI search work. It would be easier for us to sell you an llms.txt file. But the honest read of the evidence doesn't support it, so here it is.

What is llms.txt?

It's a proposed standard: a plain text file at the root of your domain, like yoursite.com/llms.txt, containing a curated Markdown summary of your site and links to your important pages. The idea is to hand AI models a clean, structured version of your content instead of making them parse your HTML.

It's a reasonable idea. That's separate from whether it works.

Do AI companies actually use it?

Mostly no, and one of them has said so directly. Google's Gary Illyes said on stage at Google Search Central Live Deep Dive that Google doesn't support llms.txt and isn't planning to (Search Engine Roundtable). Google also removed llms.txt from its Search developer documentation.

Google's John Mueller was blunter: "None of the AI services have said they're using llms.txt, and you can tell when you look at your server logs that they don't even check for it" (Search Engine Land).

There is a counterpoint worth stating. Ray Martinez reported log-file evidence of OpenAI requesting his llms.txt file, saying it was "pinging our servers every 15 minutes looking for freshness" (Search Engine Roundtable). That was a handful of sites and one person's logs, not a study, and a crawler fetching a file is not the same as a model using it. But it's real and it cuts against the flat "nobody touches it" claim.

Several AI companies, including Anthropic, do publish llms.txt files on their own documentation sites. Publishing one is not the same as parsing yours.

Has anyone tested whether it moves traffic?

Yes, and the results are underwhelming. Search Engine Land tracked 10 sites across finance, B2B SaaS, ecommerce, insurance and pet care for 90 days before and 90 days after adding llms.txt, measuring AI crawl frequency and traffic from ChatGPT, Claude, Perplexity and Gemini.

The results: eight sites saw no measurable change. One insurance site declined 19.7%. Two saw increases of 25% and 12.5%, and the authors concluded llms.txt was not the cause of either (Search Engine Land).

For the seven unchanged ecommerce and SaaS sites, the finding was specific: after 90 days, traffic was flat, crawl frequency was identical, and the content documented in llms.txt was already indexed and discoverable anyway.

Ten sites is a small sample and the decline was almost certainly unrelated. But the direction is clear enough: nobody has produced evidence of a gain.

So why does everyone recommend it?

Because it's easy to sell and easy to deliver. It's a single file. It takes an hour. It produces a deliverable you can point at, and because AI visibility is hard to measure, nobody can prove it didn't work.

That's not a conspiracy, it's just incentives. Be sceptical of any AI-visibility service whose main artifact is a text file.

What actually makes you visible in AI answers?

The unglamorous version, from the providers' own documentation.

For ChatGPT, the relevant lever is your robots.txt. OpenAI runs OAI-SearchBot to surface sites in ChatGPT's search features, and says opting out means your site won't be shown in ChatGPT search results. GPTBot is the separate crawler for model training (OpenAI). Blocking GPTBot does not remove you from ChatGPT search. Plenty of sites have blocked the wrong one.

For Google's AI features, eligibility is simpler than people assume. Google states a page needs to be indexed and eligible to show with a snippet, with no additional technical requirements, and that structured data is not required. The things that silently disqualify you are nosnippet, data-nosnippet, a restrictive max-snippet, and noindex (Google Search Central).

So the real checklist is: be crawlable, be indexed, don't suppress your own snippets, put your important content in actual text, and answer specific questions clearly. That's it. It's the same work that earns rankings.

And the queries worth targeting are narrow ones. More than 68% of terms that trigger AI Overviews get 100 or fewer monthly searches (Semrush), which favours businesses willing to answer precise questions over those chasing broad terms.

Should you add one anyway?

If it takes you an hour and costs nothing, sure. The downside is close to zero, a couple of AI vendors publish their own, and if adoption arrives you're ready.

Two conditions. Don't pay much for it, and don't let it substitute for the work above. If your site isn't crawlable and your pages don't answer real questions, a tidy text file changes nothing.

One practical warning from our own experience this week: if you do publish one, check that it actually serves plain text. We found a file written in a rich-text editor that was serving RTF control codes to anything that read it. It looked fine in a browser. It was gibberish to a parser. Open your own file and look at the raw source.

Frequently asked questions

What is llms.txt in simple terms?
A plain text file at your domain root containing a Markdown summary of your site and links to key pages, intended to help AI models understand your content without parsing HTML.

Does Google use llms.txt?
No. Gary Illyes said Google doesn't support it and isn't planning to, and Google removed it from its Search developer documentation.

Is llms.txt the same as robots.txt?
No. Robots.txt controls crawler access and is genuinely honored. llms.txt is a proposed content-summary format with no confirmed consumer. Robots.txt is the one that matters.

Will llms.txt improve my rankings?
There's no evidence it does. A 90-day study of 10 sites found eight unchanged, and the two that improved were attributed to other causes.

What should I do instead?
Make sure you're crawlable and indexed, don't block OAI-SearchBot, remove any nosnippet directives, put key content in plain text, and write pages that answer specific customer questions.

What we'd tell a client

Spend the hour on your service pages instead. If you want the file, add it at the end of a list that starts with being indexed and answering real questions.

You can run our full AI visibility audit yourself, and how to show up on ChatGPT covers the crawler side properly. If you'd rather we looked, that's what we do.

Book a 30-minute intro call and we'll tell you honestly whether AI visibility is your bottleneck or whether something else is.