AI crawler access census

← Scan your site · Trend · Signals

robots.txt is where a store decides whether AI crawlers may read it. This counts Disallow: / rules per crawler across 210 stores (150 of them publish a robots.txt).

CrawlerBlocked byShare
Amazonbot64.0%
Bytespider53.3%
CCBot53.3%
ClaudeBot53.3%
GPTBot53.3%
Applebot-Extended42.7%
Diffbot42.7%
Google-Extended42.7%
cohere-ai42.7%
omgili42.7%
Timpibot21.3%
anthropic-ai21.3%
PerplexityBot10.7%

Stores that disallow / for * (this blocks every crawler, AI or not): 1

Which store blocks which crawler

SiteBlocked crawlers
wordery.comAmazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, anthropic-ai, cohere-ai, omgili
www.amazon.comBytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, PerplexityBot, Timpibot, cohere-ai, omgili
www.nintendo.comApplebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, Timpibot, cohere-ai, omgili
www.buckmason.comAmazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, anthropic-ai, cohere-ai
www.aliexpress.comAmazonbot, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended
www.beardbrand.comAmazonbot
www.ikea.comAmazonbot
www.bol.comomgili
www.ebay.comApplebot-Extended
www.chewy.comAmazonbot
bookshop.orgDiffbot

Method and limitations (read before quoting)

Weekly digest

中文