Quick answer: if you're adding it hoping to be cited more often by ChatGPT or Google's AI answers, the evidence says no. Google states plainly that its Search systems don't use the file at all, most llms.txt files are never fetched by anything, and studies looking for a citation effect haven't found one. But the file isn't useless — it serves a different job entirely, and a few major AI companies genuinely do use it for that job. Whether you need one depends on which of the two you were hoping for.
The question arrives in a predictable form: someone read that llms.txt is "the new robots.txt", that AI systems use it to understand your site, and that adding it improves your chances of being recommended. Each part of that is either wrong or describes something other than what it sounds like.
Here's what the file is, what the measurements show, and the one case where it's worth shipping.
What an llms.txt file actually is
A markdown file at the root of your domain containing a curated map of your site — the pages that matter, with short descriptions, organised so a machine can see the structure without crawling everything.
The proposal dates from September 2024. The idea is reasonable: a large documentation site has thousands of pages, and a system reading it without guidance may index outdated material, miss what matters and misread the structure. A curated index gives it a better starting point.
Note what that describes. It's a navigation aid, not an access control and not a ranking signal. It cannot block any crawler, and it doesn't tell a system anything it couldn't otherwise find — it just makes finding it cheaper.
Does llms.txt help you get cited by AI assistants
This is the question people actually mean, and the answer is no, as far as anything measurable shows.
Google's Search Central documentation states it directly: Google Search does not use llms.txt — not for rankings, not for AI Overviews, not at all. That position has been consistent since Gary Illyes confirmed it in July 2025.
Independent measurement points the same way. Across hundreds of thousands of domains, 97% of llms.txt files were never fetched by anything, and statistical models looking for a citation effect found none. One monitoring exercise covering more than 500 million AI bot visits over a ninety-day window recorded 408 requests targeting llms.txt directly — a rounding error.
Adoption reflects that. A study of 300,000 domains put the implementation rate at just over 10%, eighteen months into an industry conversation that has been anything but quiet.
Why the file has this problem structurally
John Mueller compared llms.txt to the keywords meta tag, and the comparison is worth taking seriously rather than as dismissal.
The keywords tag failed for a specific reason: it was a self-declaration that nobody could verify. A site said what it was about, and there was nothing stopping it from saying whatever was most convenient. Search engines eventually stopped reading it entirely.
llms.txt has the same shape. It's a file in which a site describes its own importance, with no verification anywhere in the loop. That doesn't make it dishonest, but it does explain why systems that have to judge credibility at scale treat it cautiously — and it's the same reason the rest of AI visibility work rewards what can be corroborated rather than what's asserted.
Who actually reads llms.txt, and for what
Here's where the "useless" verdict overshoots, because several major AI companies do use the file — just not for the thing most people are asking about.
Anthropic recommends it in its guidance on writing for agents. OpenAI maintains llms.txt files for its Agents SDK. Perplexity has said it retrieves the file to help prioritise which pages to read. And in May 2026 Chrome's Lighthouse moved its llms.txt audit out of experimental status into a default agentic browsing category — putting the Chrome team and the Search team on visibly opposite sides.
The reconciliation is that the file serves agentic retrieval, not conversational answering.
When a developer asks a coding assistant to work against someone's documentation, the assistant can fetch that site's llms.txt as a table of contents and traverse only the pages it needs, rather than scraping the whole site. That's a real workflow, it happens constantly, and it's invisible to studies that measure crawler behaviour or citation rates — because it isn't crawling and it isn't citation.
Which explains the adoption pattern precisely. The companies shipping llms.txt are documentation-heavy: developer platforms, infrastructure providers, API companies. Marketing sites, blogs and B2B service sites mostly haven't, and the reason isn't neglect — their audience isn't coding agents.
Should your site have an llms.txt file
Three questions settle it, and they're worth answering honestly rather than optimistically.
Do agents work against your content? If you publish documentation, an API reference or technical material that developers point tools at, yes — ship one. The benefit is real and immediate for those users, whatever the citation studies say.
Are you adding it to improve AI visibility? Then no, and the effort belongs somewhere with evidence behind it: facts stated so a machine can repeat them, structured data that matches your copy, corroboration outside your own site, and content that survives being summarised. Those move the needle; this doesn't.
Do you have spare capacity and a taste for optionality? Then it's defensible. The file takes an hour, breaks nothing, and if agentic browsing grows as a traffic source, having it costs you nothing to have had. Just don't count it as work done on visibility.
For most businesses the honest answer is the second one.
The implementation mistake that actually causes damage
If you do ship one, avoid the version that turns a harmless file into a real problem.
A popular approach involves generating a markdown copy of every page on the site alongside the index. If those copies are indexable, you've created duplicate content at scale — every page now exists twice, competing with itself, diluting crawl budget across a set of files that serve no reader.
The file is supposed to be a curated index: your important pages, described briefly, linked. Not a second copy of the site. A small site can list twenty entries; a large one should list the twenty that matter rather than all nine hundred.
What actually controls AI access to your site
Worth separating, because the two files get conflated constantly and only one of them has real support.
robots.txt is the file with genuine, deliberate backing from every major AI crawler operator. OpenAI, Anthropic, Google, Perplexity and the rest publish user-agent strings and honour the directives in practice. If you want to control whether AI systems read your site at all, that's where it happens.
llms.txt cannot block anything. It's a suggestion about what to read once reading is already permitted.
The practical consequence: if AI visibility matters to you, the thing worth checking isn't whether you have an llms.txt file. It's whether your robots.txt is accidentally blocking the crawlers you want. That takes two minutes and it's the one of the two that can actually cost you.
How I'd approach llms.txt for a client site
I don't sell llms.txt implementation, and I'd be cautious of anyone who does — it's an hour of work being packaged as an AI visibility service.
Where a client's site is documentation-heavy and their users point coding tools at it, I'll add one, as a developer-experience improvement rather than a search tactic. Where it isn't, I'll say what this article says and put the time into the things that measurably affect whether an assistant names you.
And if it's added, it gets measured like anything else: a fixed set of prompts run before and after, so the question of whether it changed anything is answered rather than assumed.
Where to start if you're wondering about llms.txt
Reduced to one principle: decide which of the two jobs you wanted, because the file only does one of them.
A practical step for today: open your robots.txt and check that the AI crawlers you want reading your site aren't blocked. That's the file with real consequences, it takes two minutes, and a surprising number of sites have a directive in there that nobody remembers adding.
If you'd like to know what assistants currently say about your company — which is the question llms.txt is usually a proxy for — write to me and we'll look at it properly.
