Agentic commerce
llms.txt
llms.txt is a convention placing a markdown file at a site's root that summarises what the site is and which URL answers which question, so assistants need not crawl it all.
Whether major assistants consume it is still genuinely unsettled. What is settled is adoption: we checked 58 well-known DTC brands and 73% of the reachable ones already publish one — but 70% of those did not write it, because Shopify generates a templated file automatically. Publishing one is therefore table stakes rather than an edge.
That moves the useful question from whether to have one to whether the default describes you. A generated 4.3KB file describes a generic store; it does not know which products matter, which questions your buyers ask, or what separates you from the twenty-two other brands publishing a near-identical document.
The one rule that decides whether it is worth anything: generate it, do not hand-write it. A hand-maintained index rots within two deploys, and a stale index is worse than none because it confidently points at pages that have moved.
There is now evidence for that rule beyond staleness. Among the stores we found accepting an agent checkout, several publish a hand-authored file that never names the endpoints their own store answers on — so the file a careful team wrote hides a capability the generated one announces. Read what is at that path before replacing it.
What the file looks like
The proposal describes a markdown document rather than a new syntax to learn. It opens with a level-one heading carrying the site or project name, followed by a short blockquote saying what the site is — the one paragraph a reader with no other context needs. Ordinary prose may follow it.
Below that come level-two sections, each holding a list of links with a brief note against each one: what the page is, and when it is the right page. A section conventionally set aside for optional material marks links that can be skipped when a shorter context is needed, which is the only prioritisation the format itself carries.
A companion convention places an expanded file at a separate path, containing the content itself rather than an index of links. The two serve different needs: one orients a reader who has never seen the site, the other supplies text without navigation chrome wrapped around it.
The format is deliberately unremarkable. Everything consequential about the convention is editorial — which pages you name and how you describe them — because there is nothing else in it to get wrong.
Where it sits among the machine-facing files
The confusion that costs teams something is not about the format, it is about what the file governs. Setting it beside the other surfaces a store publishes for machines makes the boundary obvious.
The last row is the whole argument. Three of these have documented consumers and defined consequences; the fourth has neither, which is simultaneously the case for publishing one and the case against spending a project on it.
| Surface | What it states | Documented consumers | What goes wrong if it is wrong |
|---|---|---|---|
| robots.txt | Which crawlers may fetch which paths | Every major crawler documents honouring it | A careless disallow removes you from answer engines entirely |
| sitemap.xml | Which URLs exist and when they changed | Search engines document reading it | Slow or missed discovery of new and changed pages |
| schema.org JSON-LD | What the facts on this page are | Search engines document using it for rich results | A stale price or availability produces confident wrong answers |
| llms.txt | What the site is and which URL answers what | No assistant vendor has documented consuming it | Very little — which is also the honest case against effort |
What it cannot do
Three capabilities are regularly attributed to this file that it does not have, and each has a different correct home. Getting them wrong is how a team comes to believe it has set a policy when it has written a summary.
- It grants and withholds nothing. There is no directive syntax and no permission semantics, so it cannot block a crawler. Permission is robots.txt for crawlers that honour it, and an edge rule for those that do not.
- It does not replace structured data. An index of URLs says nothing about the price or availability on the page it points at. Facts belong in markup on the page itself, generated from the same source the page renders from.
- It is not a sitemap. A sitemap enumerates every URL for completeness; this file selects a few and explains them. Reproducing the sitemap here discards the only thing the format offers, which is editorial judgement about what matters.
What to check on your own domain
Most stores already serve one without anybody having decided to, because several platforms generate it. That makes auditing the file you have more useful than writing a new one, and the audit is four commands rather than a project.
- The content-type. A route quietly returning HTML looks correct in a browser and is worthless to a machine — and a catch-all page answering 200 at that path is counted as an adopter by any survey checking status codes rather than content.
- Whether it names anything you actually expose. A generated file sometimes lists machine endpoints that a hand-written replacement silently drops, so read what is there before overwriting it.
- Whether it is generated. A hand-maintained index rots within two deploys, and a stale index is worse than none because it points confidently at pages that have moved.
- Whether anything requests it. Grep the access log for the path. A file nothing fetches is not an emergency, but it is the only direct evidence available on this question and it costs one command to obtain.
Frequently asked questions
- What is llms.txt?
- A markdown file at a site's root that summarises what the site is and which URL answers which question, so an AI system can orient without crawling everything. It is a community proposal rather than a standard published by any search or assistant vendor, and it grants no permissions of any kind.
- What goes in an llms.txt file?
- A heading with the site name, a short blockquote stating what the site sells and who it serves, then sections of links with a one-line description each — categories, and the pages answering real pre-purchase questions such as sizing, shipping and returns. Point at the sitemap index for the complete URL list rather than reproducing it.
- Where does the llms.txt file go?
- At the root of the domain customers actually type, served as markdown or plain text. Check the content-type after deploying: a route that quietly returns HTML renders fine in a browser and is useless to a machine, and it is the most common way this file is silently broken.
- Do AI assistants actually read llms.txt?
- No assistant vendor has documented consuming it, and Google has stated publicly that its systems do not. Some crawler logs show the path being requested, but a fetch is not the same as a model using the content. Treat the file as cheap and unproven rather than as a technique with an expected result.
- Is llms.txt the same as robots.txt?
- No. robots.txt states which crawlers may fetch which paths and is honoured voluntarily by well-behaved crawlers. llms.txt offers a summary and grants nothing. Removing robots.txt changes what crawlers are allowed to do; removing llms.txt changes only what they are handed as a shortcut.
- What is llms-full.txt?
- A companion convention: instead of an index of links, a single file carrying the expanded content itself, so a system can read the material without fetching many pages and stripping navigation from each one. It is much larger by design and answers a different need from the index file.
In depth: Does llms.txt actually work?