About AgentBI

What this is

AgentBI measures how friendly a site is to AI agents: can an agent fetch your pages, is your product data machine-readable (JSON-LD price / availability), have you deployed a UCP manifest, do you publish llms.txt.

How we crawl

Two exceptions, explained

One step, the multi-UA probe, requests your site with several real AI agent identifiers (GPTBot, ClaudeBot, …) and one real browser identifier, then compares the responses — to answer "do you treat agents differently from normal visitors". This only measures anything if the identifiers are real, so it is a measurement, not anonymous crawling. Second: in the daily census, a site that returns non-200 to our scanner gets one extra request with a real browser identifier — to tell "this site blocks non-browser clients" apart from "our network is blocked". Every other fetch uses the honest agent above.

Don't want to be scanned

robots.txt is not honoured automatically yet — known TODO. Two ways out:

Data

Results are published on the leaderboard. Only sites returning 200 are counted; sites returning non-200 (mostly enterprise WAFs) are absent and are not read as "low agent-readiness". The run is daily and the raw result — every field, one row per store — is public at agentbi.tech/batch.json, so any number here can be recomputed.

Our own score

We point this at ourselves: 55/100 (2026-10-18), and that is our ceiling. We ship /llms.txt, a robots.txt that names and allows AI agents, a sitemap and JSON-LD — but we sell nothing, so Product structured data and UCP (45 of the 100 points) are unreachable for us. So every report also carries a readability sub-score: fetchability + llms.txt + robots, out of 55 — the same scale for a shop and for a site like ours. Ours is 55/55. The commerce suggestions (UCP, Product JSON-LD) are grouped on their own and do not apply to sites that sell nothing.

← Back to scanning · 中文