← Scan your site · Trend · Signals
robots.txt is where a store decides whether AI crawlers may read it. This counts Disallow: / rules per crawler across 210 stores (150 of them publish a robots.txt).
| Crawler | Blocked by | Share |
|---|---|---|
Amazonbot | 6 | 4.0% |
Bytespider | 5 | 3.3% |
CCBot | 5 | 3.3% |
ClaudeBot | 5 | 3.3% |
GPTBot | 5 | 3.3% |
Applebot-Extended | 4 | 2.7% |
Diffbot | 4 | 2.7% |
Google-Extended | 4 | 2.7% |
cohere-ai | 4 | 2.7% |
omgili | 4 | 2.7% |
Timpibot | 2 | 1.3% |
anthropic-ai | 2 | 1.3% |
PerplexityBot | 1 | 0.7% |
Stores that disallow / for * (this blocks every crawler, AI or not): 1
| Site | Blocked crawlers |
|---|---|
| wordery.com | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, anthropic-ai, cohere-ai, omgili |
| www.amazon.com | Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, PerplexityBot, Timpibot, cohere-ai, omgili |
| www.nintendo.com | Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, Timpibot, cohere-ai, omgili |
| www.buckmason.com | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, anthropic-ai, cohere-ai |
| www.aliexpress.com | Amazonbot, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended |
| www.beardbrand.com | Amazonbot |
| www.ikea.com | Amazonbot |
| www.bol.com | omgili |
| www.ebay.com | Applebot-Extended |
| www.chewy.com | Amazonbot |
| bookshop.org | Diffbot |
Disallow: / rules matched to 13 known AI crawler tokens in each store's robots.txt. Snapshot 1795376326.