Agentic commerce
How do you check whether AI crawlers are visiting your site?
Updated
Grep your access logs for the user-agent strings of the three crawler classes and count them separately: training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider), answer engines (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and live agent fetches (ChatGPT-User, Perplexity-User). Seeing zero answer-engine hits over a week is the finding that matters — it almost always means a CDN or WAF rule is refusing them before your origin ever logs the request.
What to grep for
Count the three classes separately. A single total tells you nothing useful, because the classes have opposite implications: training crawls are neutral, answer-engine crawls are how you get surfaced, and live agent fetches mean a person is asking about you right now.
- Training: GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, Applebot-Extended, meta-externalagent
- Answer engines: OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot
- Live agent fetches: ChatGPT-User, Perplexity-User
Where to look, in order of usefulness
Your origin access log is the obvious place and the least reliable one, because anything blocked at the edge never reaches it. Check the CDN log first if you have one — Cloudflare, Fastly, CloudFront and Akamai all expose request logs with user-agent, and crucially they show requests that were refused as well as served.
That difference is the whole point of the exercise. An origin log showing no OAI-SearchBot traffic and a CDN log showing OAI-SearchBot receiving 403s describe completely different problems, and only the second one is actionable by you.
Reading the result
Three patterns, three conclusions.
| What you see | What it means | What to do |
|---|---|---|
| Answer engines crawling, 200s | You are reachable. Whether you are cited depends on your content and machine readability. | Check what they can actually parse — price and availability in raw HTML. |
| Answer engines getting 403 or 429 | A bot-management rule is refusing them. This is the common case and it is invisible in robots.txt. | Allowlist them at the CDN. Nothing else you do matters until this is fixed. |
| No answer-engine requests at all | Either they were refused before your origin logged anything, or the site is not in their crawl frontier yet. | Check the CDN log. If it is clean there too, the issue is discovery — sitemap, links, indexing. |
Verify rather than trust the user-agent
A user-agent string is self-reported and trivially forged, so a log full of "GPTBot" hits is not proof that OpenAI visited. Scrapers impersonate reputable crawlers precisely because so many sites allowlist them.
The operators publish IP ranges for their crawlers, and a reverse DNS lookup on the requesting IP is the standard check. This matters most when you are deciding whether crawl load justifies a block: throttling a forged GPTBot is fine, and throttling the real one is a policy decision you should make deliberately.
The other half of the picture
Log analysis tells you whether crawlers can reach you. It does not tell you whether what they found was parseable — and the second gap is larger industry-wide. Adobe measured retail product pages scoring 66% on machine readability, the lowest of any page type.
Our free checker covers that half: it fetches a page with no JavaScript executed and reports what an agent can actually read, and it reads your robots.txt and reports the three crawler classes separately. It cannot see your CDN rules — nothing external can, which is why the log check above is worth doing by hand.
Frequently asked questions
- How often should AI crawlers visit?
- There is no benchmark worth quoting, because it varies enormously with site size, update frequency and existing authority. The signal to act on is not the rate but the status codes: answer-engine requests receiving 403s is a problem at any volume, and zero requests over a week is worth investigating.
- I see no AI crawler traffic at all. Is my site blocked?
- Check the CDN log before concluding anything. Requests refused at the edge never reach your origin, so an empty origin log looks identical whether you are being blocked or simply have not been discovered. Those need opposite fixes.
- Can I trust the user-agent string?
- No — it is self-reported and commonly forged, precisely because reputable crawler names get allowlisted. Verify against the operator's published IP ranges or by reverse DNS on the requesting address before making a decision based on volume.
- Does ChatGPT-User traffic mean someone is buying?
- It means someone asked an assistant a question that led it to fetch your page for them, in that moment. It is the closest thing to a live intent signal in this whole category, and it is worth separating from indexing traffic in any report you build.
Sources
- AI traffic grows but retail sites lag in AI search visibility (Q1 2026) — Adobe Digital Insights