robots.txt 是一家店决定「要不要让 AI 爬虫读」的地方。这里统计同一批 210 个站点(其中 150 个提供了 robots.txt)里,各爬虫被 Disallow: / 的次数。
| 爬虫 | 被封锁于 | 占比 |
|---|---|---|
Amazonbot | 6 | 4.0% |
Bytespider | 5 | 3.3% |
CCBot | 5 | 3.3% |
ClaudeBot | 5 | 3.3% |
GPTBot | 5 | 3.3% |
Applebot-Extended | 4 | 2.7% |
Diffbot | 4 | 2.7% |
Google-Extended | 4 | 2.7% |
cohere-ai | 4 | 2.7% |
omgili | 4 | 2.7% |
Timpibot | 2 | 1.3% |
anthropic-ai | 2 | 1.3% |
PerplexityBot | 1 | 0.7% |
有 1 个站对 * 写了 Disallow: /——那会拦掉所有爬虫,不只是 AI。
| 站点 | 被封锁的爬虫 |
|---|---|
| wordery.com | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, anthropic-ai, cohere-ai, omgili |
| www.amazon.com | Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, PerplexityBot, Timpibot, cohere-ai, omgili |
| www.nintendo.com | Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, Timpibot, cohere-ai, omgili |
| www.buckmason.com | Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, anthropic-ai, cohere-ai |
| www.aliexpress.com | Amazonbot, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended |
| www.beardbrand.com | Amazonbot |
| www.ikea.com | Amazonbot |
| www.bol.com | omgili |
| www.ebay.com | Applebot-Extended |
| www.chewy.com | Amazonbot |
| bookshop.org | Diffbot |
Disallow: / 规则。快照日期 1795376326。