Can llms.txt block AI crawlers?

No. It has no directive syntax and no permission semantics. Blocking is expressed in robots.txt for crawlers that honour it, and enforced at your edge for those that do not.

You need robots.txt, and you should know what it currently says. llms.txt is optional and cheap; add it once the permission file is deliberate rather than inherited.

Does robots.txt need a separate rule for each AI crawler?

It needs as many rules as you have distinct intentions. Training crawlers, answer-engine crawlers and live agents fetching on behalf of a user are three different things, and most sites want different answers for each.

Agentic commerce

What is the difference between llms.txt and robots.txt?

Updated August 14, 2026

robots.txt states which crawlers may fetch which paths; it is a permission file that well-behaved crawlers honour voluntarily. llms.txt offers a summary of what a site is and which URL answers which question; it grants and withholds nothing. Removing robots.txt changes what crawlers are allowed to do. Removing llms.txt changes only what they are handed as a shortcut. Neither file enforces anything — enforcement happens at your edge.

The distinction in one line each

They are confused constantly because they share a location and a file extension, and the practical consequence of confusing them is that teams believe they have set a policy when they have written a summary.

	robots.txt	llms.txt
Purpose	Permissions	Orientation
Format	Directives	Markdown
Standardised	Yes, long-established	A proposal, widely adopted
Honoured by	Well-behaved crawlers	Unconfirmed
Affects crawling	Yes	No
Enforces anything	No	No

The two root files compared on the dimensions that decide how you use them.

Neither one is enforcement

This is the point most guides skip. Both files are requests. A crawler that ignores robots.txt fetches your pages anyway, and nothing in either file stops it.

Real enforcement lives at the edge: WAF rules, rate limits, bot management. That distinction is not academic. When we surveyed DTC brands, the sites that refused our requests refused at the edge while their robots.txt permitted everything — so an audit of the file alone would have reported the opposite of what actually happened.

Which one to get right first

robots.txt, without hesitation. It is the file with consequences: a careless disallow can remove you from answer engines entirely, and the crawlers it governs are named, documented and separable into training, answer-engine and live-agent classes that deserve different answers.

llms.txt can be generated afterwards in an afternoon. The ordering follows from the asymmetry — one file can cost you visibility, the other can at best hand a reader a shortcut.

Frequently asked questions

Can llms.txt block AI crawlers?: No. It has no directive syntax and no permission semantics. Blocking is expressed in robots.txt for crawlers that honour it, and enforced at your edge for those that do not.
Do I need both?: You need robots.txt, and you should know what it currently says. llms.txt is optional and cheap; add it once the permission file is deliberate rather than inherited.
Does robots.txt need a separate rule for each AI crawler?: It needs as many rules as you have distinct intentions. Training crawlers, answer-engine crawlers and live agents fetching on behalf of a user are three different things, and most sites want different answers for each.

Sources

DTC AI crawler index — 147 brands, collected 14 August 2026 — PlatformDTC (primary data)
The /llms.txt file — proposal and specification — llmstxt.org (primary specification)