AI 爬虫准入普查

← 检测我的站点 · 趋势 · 信号

robots.txt 是一家店决定「要不要让 AI 爬虫读」的地方。这里统计同一批 210 个站点(其中 150 个提供了 robots.txt)里,各爬虫被 Disallow: / 的次数。

爬虫被封锁于占比
Amazonbot64.0%
Bytespider53.3%
CCBot53.3%
ClaudeBot53.3%
GPTBot53.3%
Applebot-Extended42.7%
Diffbot42.7%
Google-Extended42.7%
cohere-ai42.7%
omgili42.7%
Timpibot21.3%
anthropic-ai21.3%
PerplexityBot10.7%

有 1 个站对 * 写了 Disallow: /——那会拦掉所有爬虫,不只是 AI。

哪些站封锁了哪些爬虫

站点被封锁的爬虫
wordery.comAmazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, anthropic-ai, cohere-ai, omgili
www.amazon.comBytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, PerplexityBot, Timpibot, cohere-ai, omgili
www.nintendo.comApplebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot, GPTBot, Google-Extended, Timpibot, cohere-ai, omgili
www.buckmason.comAmazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, anthropic-ai, cohere-ai
www.aliexpress.comAmazonbot, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended
www.beardbrand.comAmazonbot
www.ikea.comAmazonbot
www.bol.comomgili
www.ebay.comApplebot-Extended
www.chewy.comAmazonbot
bookshop.orgDiffbot

方法与局限(引用前请先读)

每周摘要

English