About AgentBI
Agent-scannable intelligence for cross-border DTC sites.
What this is
AgentBI measures how friendly a site is to AI agents: can an agent fetch your pages, is your product data machine-readable (JSON-LD price / availability), have you deployed a UCP manifest, do you publish llms.txt. The criteria are public in a 100-item checklist; this site does not re-judge them.
How we crawl
- Only when someone submits a URL. No continuous crawling of the web.
- The user agent is honest and identifiable: agentbi/0.1 (+https://agentbi.tech/about; agent-readiness scanner) — not disguised as a browser.
- One check makes about 10 requests to a site and takes 15-20 seconds.
- One real scan per domain per 10 minutes; results are cached for 30 minutes.
Two exceptions, explained
The multi-UA probe requests your site with several real AI agent identifiers and one real browser identifier, then compares the responses — to answer whether you treat agents differently from normal visitors. This only measures anything if the identifiers are real, so it is a measurement, not anonymous crawling. Second: in the daily census, a site that returns non-200 to our scanner gets one extra request with a real browser identifier — to tell a site that blocks non-browser clients apart from our network being blocked. Every other fetch uses the honest agent above.
Data
Every day at 04:30 a fixed observer egress scans the main pool and compares each field with the previous snapshot; the raw result (one row per site, all fields) is public at agentbi.tech/batch.json, so any number here can be recomputed. Every number is labelled with its vantage (egress): geo redirects, regional catalogs and local-currency pricing make one site answer differently per egress.
Don't want to be scanned
Automatic robots.txt compliance is still a known TODO. Two ways out today:
- (Preferred) Open an issue with the stop-scanning template and we add the domain to a blocklist checked before fetching.
- Or email contact@agentbi.tech with stop and the domain is added to the blocklist.
Our own score
We hold ourselves to the same checklist: M1 ships llms.txt, a robots.txt that names individual agents, a sitemap, structured data and /.well-known/ucp. But the current self-scan has to run against a public address after deployment — the local scanner refuses non-public addresses by design (SSRF guard), so we do not publish a number cobbled together locally. Historical record: 55/100 measured on the old site on 2026-10-18, for the record only.