DTC AI crawler index

Method and limits

One HTTPS GET each to /robots.txt and /llms.txt per domain, redirects followed, user-agent PlatformDTC-AgentReadiness/1.0 with a contact URL, no JavaScript executed, no retries. Rules are resolved per agent with the agent-specific group taking precedence over the wildcard group, and consecutive User-agent lines treated as one shared rule block. An llms.txt counts only if it returns 200 with more than 20 bytes and a non-HTML content type; “generated” means it matches the fixed # Agent Instructions — <Brand> opening that Shopify emits.

Two limits worth carrying into any citation. Our user-agent is unrecognised by every bot-management vendor, so a refusal shows a site declines unknown crawlers — not that it declines OAI-SearchBot, which vendors commonly allowlist. And this is a convenience sample of well-known US DTC brands that skews heavily to Shopify, which is precisely the platform generating those llms.txt files; a sample of enterprise retailers would look very different.

This is a dated snapshot, not a live crawl. That is deliberate — a page that re-crawls 147 third-party domains on demand is a bad neighbour, and a number that silently changes underneath a citation is worse than one that is openly out of date.

The write-ups

• We checked 30 DTC brands’ robots.txt. None block AI answer engines.
• 73% of DTC brands have an llms.txt. Almost none of them wrote it.
• We scored 23 DTC product pages. Machine readability is almost binary.
• Run the same checks on your own store