llms.txt Explained — What It Is, Who Reads It, and Whether You Need One
What llms.txt is, whether answer engines actually read it, and the SPA fallback bug that makes most implementations worse than having no file.
llms.txt is a proposed standard: a markdown file at the root of your domain that describes your site and links to its most important pages, written for AI clients rather than browsers. Adoption by the major engines is limited and largely unconfirmed, so treat it as cheap insurance rather than a ranking lever.
That is the honest summary. Below is what it is for, what it is not, and the implementation detail that silently breaks it on most single-page applications.
What it looks like
A minimal, valid file:
# Your Company
> One-paragraph description of what you do, written so it can be quoted
> without surrounding context.
## Pages
- [Pricing](https://example.com/pricing): What each plan includes.
- [Documentation](https://example.com/docs): Setup and API reference.
## Articles
- [A guide to X](https://example.com/blog/x): What the article establishes.
## Contact
- Email: hello@example.com
The format is deliberately plain: an H1 with your name, a blockquote summary, then H2 sections of annotated links. The annotations matter — a bare list of URLs carries almost no information, while a one-line description of each page tells a retrieval system what it would be fetching.
What it is for
The argument behind it is reasonable. An AI client that wants to understand your site has to crawl it, parse navigation written for humans, and infer which pages matter. llms.txt short-circuits that: here is what we do, here are the pages worth reading, in a format that needs no parsing.
It is a curation signal, not an access-control file. Which brings us to the most common confusion.
It is not robots.txt
They solve opposite problems:
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | What crawlers may not fetch | What clients should read |
| Tone | Restriction | Recommendation |
| Format | Directives | Markdown |
| Status | Universally honoured | Proposed, patchily adopted |
| Enforced | Yes, by convention | No |
llms.txt grants no permissions and blocks nothing. If you want to control AI crawler access, that is still robots.txt.
Does anything actually read it?
Be sceptical of confident claims either way. No major engine has committed publicly to using it as a ranking or retrieval input, and there is no reliable way to observe one doing so from the outside.
What is true: some AI developer tools and documentation consumers read it, it costs almost nothing to produce, and it does no harm. That is a reasonable basis for shipping one. It is not a reasonable basis for the "essential for AI SEO" framing you will find on tool vendors' blogs — several of which sell generators.
Ship it. Do not expect it to move anything on its own. Spend the time you save on content structure and off-page presence, which demonstrably matter.
The mistake that makes it worse than nothing
This is the part worth the article, and we hit it ourselves.
Most single-page applications are served with a catch-all fallback — every unmatched path returns index.html so client-side routing works. In nginx:
location / {
try_files $uri $uri/ /index.html; # the problem
}
Now request a file you never created:
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/llms.txt
# 200
HTTP 200. Not a 404 — a success, serving your HTML shell, with Content-Type: text/html. An AI client that fetches /llms.txt receives a page of HTML, may cache it as the canonical contents of that file, and has learned nothing except that your site returns nonsense.
This is strictly worse than not having the file. A 404 is unambiguous and the client moves on. A 200 of unrelated HTML is a lie it may keep.
Check yours:
curl -s https://yoursite.com/llms.txt | head -c 100
If that begins with <!doctype html>, you have the bug — and you have it for every non-existent path on the domain, which is also unlimited soft-404 crawl space.
The fix is to serve real files as files and stop the blanket fallback:
location = /llms.txt {
default_type text/plain;
try_files $uri =404;
}
location / {
try_files $uri $uri/index.html =404;
}
Writing one worth reading
If you ship it, make the content earn its place.
Lead with a quotable summary. The blockquote under your H1 is the single passage most likely to be lifted verbatim. Write it as a standalone definition: what you are, who it is for, what is distinctive. Not a tagline.
Annotate every link. - [Pricing](url): What each plan includes and who it suits. is useful. - [Pricing](url) is not.
Curate hard. Twenty pages that matter, not two hundred. The value is the selection; a dump of your sitemap discards it.
Disambiguate your name if it collides. If your brand is routinely confused with another, say so plainly — "Spelled A-C-M-E; unrelated to Acme Corp, the logistics company." Entity confusion is a real and common problem, and this is one of the few places you can address it in a machine-readable way.
Generate it from your real routes. A hand-maintained copy goes stale within two releases. OPSYRA's is emitted by the build from the same route registry that produces the sitemap, so a new article appears in both automatically.
Where it sits in priority
Honest ordering for a site starting from scratch:
- Make sure crawlers can read your pages at all — raw HTML with real text.
- Correct canonical tags per route.
- Answer-first content structure.
ArticleandFAQPageschema with visible bylines.- Off-page presence on the domains engines already cite.
llms.txt.
It is last for a reason, and it belongs on the list for a reason: fifteen minutes of work, generated automatically thereafter, with a small chance of mattering later. Just do not let a vendor sell it to you as the first item.
FAQ
Is llms.txt an official standard?
No. It is a community proposal, introduced in 2024, and it has not been ratified by any standards body or formally adopted by the major AI companies. It is a convention some sites follow and some tools read, not a specification with guaranteed support.
Do ChatGPT and Perplexity read llms.txt?
Neither OpenAI nor Perplexity has publicly confirmed using llms.txt as a retrieval or ranking input, and there is no reliable external way to verify it. Some AI developer tools and documentation consumers do read it. Treat support as unconfirmed and implement it because it is cheap, not because it is proven.
Where should llms.txt be placed?
At the root of your domain, at https://yoursite.com/llms.txt, served as plain text with a text/plain content type. Verify it returns the actual file rather than your HTML shell, since single-page applications with a catch-all route fallback commonly return the app's index.html for it with a 200 status.
What is the difference between llms.txt and robots.txt?
robots.txt controls what crawlers are permitted to fetch and is universally honoured. llms.txt describes and curates your content for AI clients, granting no permissions and blocking nothing, and is a proposal with limited adoption. They serve opposite purposes and one does not substitute for the other.
Will llms.txt improve my AI search rankings?
There is no evidence that it does on its own. It may help an AI client understand your site once it is already reading it, but it does not cause retrieval and does not act as a ranking signal. Content structure, authority and third-party mentions determine citations; llms.txt is a low-cost addition, not a lever.