Does blocking GPTBot hurt my Google rankings?

No. GPTBot is a model-training crawler and has no bearing on Google Search rankings or on whether you appear in AI answers. The crawler that determines AI answer visibility is OAI-SearchBot, which is a different agent with a different name.

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

All three are OpenAI and they do different jobs. GPTBot gathers content for model training. OAI-SearchBot builds the search index ChatGPT answers from. ChatGPT-User is a live fetch on behalf of a specific person who is asking about you right now. A merchant generally wants to block at most the first.

How do I check whether I am blocking AI crawlers?

Read your robots.txt, then check your CDN or WAF bot-management rules — that second place is where most accidental blocks live and where nobody thinks to look. Our free agent-readiness checker reads robots.txt for you and reports the three classes separately.

Will blocking AI crawlers stop my content being scraped?

No. robots.txt is voluntary: it is honoured by the major AI companies — the ones bringing you referral traffic — and ignored by the scrapers you are actually worried about. It reliably costs you the visibility and unreliably prevents the scraping.

Is there any case for a merchant blocking answer engines?

Very few. Perhaps a brand under an exclusivity agreement, or one whose catalog is genuinely confidential. For an ordinary store it means opting out of a channel where referred visitors have been measured converting better than non-AI traffic, in exchange for protecting copy written to be read.

Agentic commerce

Should an ecommerce store block AI crawlers in robots.txt?

Updated August 14, 2026

Block model-training crawlers if you want to; it costs nothing in search rankings or AI answer visibility. Do not block answer-engine crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, or live agent fetches like ChatGPT-User — those are what put you in AI answers and bring referral traffic. The distinction matters because around a quarter of ecommerce and B2B SaaS sites block major LLM crawlers unintentionally, often at the CDN rather than in robots.txt.

Most advice on this is written for publishers

The case for blocking AI crawlers was made by and for publishers, and on its own terms it is a strong case: if your product is the text, a model trained on that text competes with you, and you are handing over the inventory for free.

A merchant is in the opposite position. Your product is a product. The text on your site exists to sell it. Blocking the crawlers that would surface that product to a shopper who described exactly what they wanted is not protecting an asset — it is declining a channel to protect a sales page.

That is why applying publisher advice to a store consistently produces the wrong answer, and why the question worth asking is not "block or allow" but "which of the three, and for what".

Three crawlers, three different bargains

"AI crawler" names three distinct things. They arrive from the same companies, sometimes over the same infrastructure, and they want completely different things from you.

The names are deliberately confusable: GPTBot trains, OAI-SearchBot indexes for answers, ChatGPT-User is a person's assistant fetching your page. All three are OpenAI.
A rule written to block "OpenAI" that catches all three is the single most common way a merchant loses AI visibility without meaning to.

Class	Examples	What it does	Cost of blocking it
Model training	GPTBot, ClaudeBot, CCBot, Google-Extended	Collects content into training corpora	None to your rankings or AI visibility. A legitimate commercial choice.
Answer engine	OAI-SearchBot, PerplexityBot, Claude-SearchBot	Builds the index assistants answer from	Removes you from AI answers. This is the expensive one.
Live agent	ChatGPT-User, Perplexity-User	Fetches your page right now, for a specific shopper	Blocks a real customer mid-purchase. Never correct for a merchant.

The three classes and what blocking each one actually costs

The mistake is usually an accident, and usually not in robots.txt

Around a quarter of B2B SaaS and ecommerce sites block major LLM crawlers without intending to. The most commonly cited instance is OAI-SearchBot, and it is also among the highest-impact.

The reason it goes unnoticed is that it frequently is not in robots.txt at all. It is a bot-management rule at the CDN, enabled as a sensible-sounding default by someone protecting the origin from scrapers, months before anyone thought about AI referral traffic. Nobody edits a file, so nobody remembers doing it.

This is worth checking directly rather than assuming, because the failure is completely silent: there is no error, no drop in a dashboard you already watch, just an absence from a channel you cannot see.

A policy that is actually defensible

The version most merchants should land on, stated as rules rather than principles.

Allow answer-engine crawlers unconditionally. OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot. These are the channel.
Allow live agent fetches unconditionally. ChatGPT-User and Perplexity-User are a customer, arriving with intent, through a different front door.
Decide on training crawlers deliberately. GPTBot, ClaudeBot, CCBot, Google-Extended. Blocking them is free of ranking cost; allowing them may help models know your brand exists. Either answer is defensible — an accidental one is not.
Name agents individually rather than relying on a wildcard. A broad rule is how the three classes get conflated.
Check the CDN as well as robots.txt. That is where the accidental blocks live.
Remember robots.txt is voluntary. It is honoured by the companies whose traffic you want and ignored by the scrapers you are worried about, so it is a poor tool for the abuse problem and a precise one for the visibility problem.

What blocking does not solve

If the actual concern is scraping load or price harvesting, robots.txt is the wrong instrument — the operators causing that problem do not read it. Rate limiting, bot management and authentication are the tools that work there, and they can be applied without touching the crawlers that bring you demand.

And the constraint on agent-sourced revenue for most merchants is not access at all. Adobe measured retail product pages scoring 66% on machine readability, the lowest of any page type, while AI-sourced traffic to those sites grew 393% year over year. Far more merchants are invisible because their price only exists after JavaScript runs than because they blocked anyone.

You can check both at once: our free checker fetches your page with no JavaScript, and reads your robots.txt to report which of the three crawler classes you are actually blocking.

Frequently asked questions

Does blocking GPTBot hurt my Google rankings?: No. GPTBot is a model-training crawler and has no bearing on Google Search rankings or on whether you appear in AI answers. The crawler that determines AI answer visibility is OAI-SearchBot, which is a different agent with a different name.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?: All three are OpenAI and they do different jobs. GPTBot gathers content for model training. OAI-SearchBot builds the search index ChatGPT answers from. ChatGPT-User is a live fetch on behalf of a specific person who is asking about you right now. A merchant generally wants to block at most the first.
How do I check whether I am blocking AI crawlers?: Read your robots.txt, then check your CDN or WAF bot-management rules — that second place is where most accidental blocks live and where nobody thinks to look. Our free agent-readiness checker reads robots.txt for you and reports the three classes separately.
Will blocking AI crawlers stop my content being scraped?: No. robots.txt is voluntary: it is honoured by the major AI companies — the ones bringing you referral traffic — and ignored by the scrapers you are actually worried about. It reliably costs you the visibility and unreliably prevents the scraping.
Is there any case for a merchant blocking answer engines?: Very few. Perhaps a brand under an exclusivity agreement, or one whose catalog is genuinely confidential. For an ordinary store it means opting out of a channel where referred visitors have been measured converting better than non-AI traffic, in exchange for protecting copy written to be read.

Sources

Should you block AI crawlers on your ecommerce store? — Nostra
AI traffic grows but retail sites lag in AI search visibility (Q1 2026) — Adobe Digital Insights