Agentic commerce
Does blocking AI crawlers hurt your Google rankings?
Updated
Blocking model-training crawlers such as GPTBot, ClaudeBot and CCBot has no effect on Google Search rankings. Google does not use those crawls; its index comes from Googlebot, a different agent. What blocking can cost you is presence in AI answers, and only if you block the wrong class — answer-engine crawlers like OAI-SearchBot and PerplexityBot. Those two outcomes are frequently conflated, and conflating them is how a training-crawler decision accidentally removes a brand from a growing referral channel.
Two indexes, two crawlers, one common mistake
Google Search ranks what Googlebot crawled. OpenAI's training corpus is what GPTBot crawled. ChatGPT search answers from what OAI-SearchBot indexed. These are separate crawls feeding separate systems, and blocking one has no bearing on the others.
So "will blocking AI crawlers hurt my SEO" has a clean answer for the classical sense of SEO: no. Disallow GPTBot and your Google rankings are unaffected, because Google was never reading that crawl.
The question people usually mean, though, is broader — will it cost me traffic. There the answer depends entirely on which agent the rule catches, and the names make that easy to get wrong.
| Blocking this | Google rankings | AI answer visibility |
|---|---|---|
| GPTBot, ClaudeBot, CCBot (training) | No effect | No effect |
| Google-Extended (Google training) | No effect | No effect on Search; affects AI training use |
| OAI-SearchBot, PerplexityBot (answer engines) | No effect | Removes you from those answers |
| ChatGPT-User, Perplexity-User (live fetch) | No effect | Blocks a live customer lookup |
| Googlebot | Deindexes you | Indirect |
What the measured data says
Blocking is rarer than the discussion suggests. A 2025 peer-reviewed study of 40,455 consistently popular websites found roughly 8–10% fully blocking at least one AI crawler, rising to 12–14% among the largest sites — and the most frequently restricted were GPTBot and CCBot, which are training crawlers. That is the deliberate, cost-free choice rather than the expensive mistake.
In our own sample of 147 DTC brands, two of those that served robots.txt blocked answer engines and six blocked training crawlers, with the blocked set varying brand by brand rather than following a template. A separate snapshot of 15 large retailers found only Amazon and eBay restricting AI crawlers at all.
Three samples, three methods, the same shape: where blocking happens it is mostly aimed at training, and mostly deliberate.
The failure mode that does cost you
Not robots.txt. In our study, four of thirty brands refused an identified crawler outright with a 403 or 429, and three failed to connect — none of which appears in a robots.txt audit, because the decision lives in a bot-management ruleset at the CDN.
That is where the expensive version of this happens: someone protecting the origin from scrapers adds a rule years before AI referral traffic is a consideration, and nobody who would notice the missing traffic has access to the console where it was set.
If you want a real answer about your own site, read the CDN log rather than the file. An origin log showing no OAI-SearchBot and a CDN log showing OAI-SearchBot receiving 403s are completely different problems, and only the second is yours to fix.
Frequently asked questions
- Will blocking GPTBot reduce my Google traffic?
- No. Google Search ranks what Googlebot crawled; GPTBot feeds OpenAI's training corpus and is not an input to Google's index. The two are unrelated systems.
- Does blocking Google-Extended hurt my Google rankings?
- No. Google-Extended controls whether your content is used for Google's generative AI training, and Google has stated it is not a ranking signal for Search. It is a separate decision from Googlebot access.
- So what is the actual risk of a restrictive robots.txt?
- Catching answer-engine crawlers or live agent fetches in a rule you wrote for training crawlers or scrapers. That removes you from AI answers, which is a growing referral channel where traffic has been measured converting better than non-AI sources.
- How do I check what I am actually blocking?
- Read robots.txt and your CDN or WAF bot rules, because most accidental blocks live in the second and not the first. Our free checker reads robots.txt and reports training, answer-engine and live-agent crawlers separately; nothing external can see your CDN rules.