We Checked 30 DTC Brands’ robots.txt. None Block AI Answer Engines.

The number everyone repeats is that around a quarter of ecommerce and B2B SaaS sites block major LLM crawlers, usually by accident. We repeated it too, on our own crawler policy page. Then we built a checker that reads robots.txt, and it seemed worth pointing at real stores rather than citing someone else.

We checked 30 well-known direct-to-consumer brands on 14 August 2026. In robots.txt, not one of them blocks AI answer engines.

What we actually found

Of the 30 domains, 23 served robots.txt to an identified crawler. Among those 23:

• 0Block AI answer engines (OAI-SearchBot, PerplexityBot, Claude-SearchBot)
• 0Block live agent fetches (ChatGPT-User, Perplexity-User)
• 1Block model-training crawlers only

The single brand blocking anything was Warby Parker, which disallows CCBot and Bytespider — model-training crawlers — while leaving the answer engines open. That is not a mistake. It is precisely the policy we would recommend: training crawlers are a commercial choice with no ranking cost, answer engines are the channel.

The blocking is real, it just is not in robots.txt

Here is the part that reconciles our finding with the number we started with. Seven of the 30 never got as far as robots.txt at all.

Four returned a 403 or 429 to a plainly identified, single, polite request. Three failed to connect at all. None of that appears in a robots.txt audit, because there is no robots.txt in the answer — the decision was made at the CDN or WAF, by someone protecting the origin from scrapers, quite possibly years before anyone in the building thought about AI referral traffic.

So the headline advice — go and check your robots.txt — is aimed at the wrong file. Among brands of this size, robots.txt is fine. The edge is where the door gets shut.

How this sits against the other available data

Two other datasets exist and both point the same way. A June 2026 snapshot of 15 large retail domains found only Amazon and eBay restricting AI crawlers — a 16.7% block rate, with Nike, Walmart, Target, Best Buy, Home Depot, Wayfair, IKEA, Nordstrom and Macy’s all allowing every tracked crawler. And a 2025 peer-reviewed study of 40,455 consistently popular websites found roughly 8–10% fully blocking at least one AI crawler, rising to 12–14% among the largest sites, with GPTBot and CCBot the most frequently restricted.

Note what the third number is measuring: GPTBot and CCBot are training crawlers. Sites blocking those are mostly making the deliberate, cost-free choice — not the expensive mistake of shutting out answer engines.

Our contribution is the population. The 15-domain snapshot is a large-retail benchmark and the 40,455-site study is the whole web; neither is a DTC sample. Ours is 30 DTC brands, and it agrees with both while being about the segment the advice is usually aimed at.

What this does not prove

One limitation matters more than the rest, and it cuts against our own headline. Our crawler identifies itself as PlatformDTC-AgentReadiness, which no bot-management vendor has ever heard of. A 403 to us is evidence that a site refuses unrecognised crawlers. It is not evidence that the same site refuses OAI-SearchBot, which most vendors allowlist by default.

So read the 403s as an upper bound on edge blocking, not a measurement of it. What they do establish is narrower and still useful: for those four brands, whether an AI crawler gets through is decided by a bot-management allowlist that nobody in marketing can see, rather than by a file anyone can read.

The other caveats are ordinary. Thirty brands is a convenience sample of well-known US names, not a random one; larger brands have more sophisticated edge infrastructure than the median store, so this likely overstates edge blocking and understates robots.txt mistakes across the long tail. Every request came from one IP in AWS us-east-1 at one moment. Connection failures may be geographic or transient rather than deliberate.

What to do with it

If you are a DTC brand, the useful conclusion is to stop auditing the file and start auditing the edge. Ask whoever administers your CDN which crawler user-agents are allowlisted, and specifically whether OAI-SearchBot, PerplexityBot and ChatGPT-User are on that list. That question has a definite answer and almost nobody has asked it.

You can check the file half in about ten seconds with our free agent-readiness checker, which reads robots.txt and reports the three crawler classes separately — training, answer engine and live agent — because conflating them is how the expensive mistake gets made. It cannot see your CDN rules. Nothing external can, which is rather the point.

Method

One HTTPS GET to /robots.txt per domain on 14 August 2026, from AWS us-east-1, user-agent PlatformDTC-AgentReadiness/1.0 with a contact URL, following up to six redirects, no JavaScript executed, no retries. Rules were resolved per agent with the agent-specific group taking precedence over the wildcard group, and consecutive User-agent lines treated as one shared rule block. Domains: Allbirds, Warby Parker, Glossier, Casper, Away, Bombas, Chubbies, Mejuri, Ritual, Caraway, Brooklinen, Quip, Harry’s, Hims, Olipop, Liquid I.V., AG1, Magic Spoon, Native, Dr. Squatch, Fenty Beauty, SKIMS, Gymshark, Vuori, Oura, Peloton, Purple, Tushy, HelloFresh, Blue Apron.

One methodological note we owe the reader: our first pass reported eight unreachable domains rather than seven, because the checker gave up after three redirects and two of these sites legitimately use more. We raised the limit and re-ran. The numbers above are from the second pass.

The data

Every figure above comes from a published, per-brand dataset you can check rather than take on trust: the DTC AI crawler index. Free to cite with attribution. Related reading: 73% of DTC brands have an llms.txt — almost none wrote it.