About our crawler
The AI visibility check fetches a website when someone asks for a report on it. We also prepare reports on businesses we intend to contact: those scans are started from our own list by a timer, at most 1 per site per UTC day. The crawler has no other schedule.
How it identifies itself
User agent: AIVisibilityBot/1.0 (+https://aivisibilityfree.com/bot)
Robots token: AIVisibilityBot. The token is permanent: if the user agent is ever renamed, rules written for AIVisibilityBot keep applying.
What it fetches
- Once per scan it reads
/robots.txtof the scanned origin. TheAIVisibilityBotgroup, or the*group if there is none, decides every page and sitemap the scan requests, pages onwww.and other subdomains of the site included. The one exception is a redirect of the homepage, below. - If robots.txt cannot be fetched (a network error, a timeout, or a 408, 429 or 5xx answer), the scan fetches nothing else. A 404, 410 or other 4xx answer, or a body that is not robots rules, counts as no rules.
- Then it reads the homepage, the sitemaps, and other pages of the same site: at most 15 pages per scan, the homepage included. Sitemaps: at most 3 named in robots.txt (else
/sitemap.xml), and for a sitemap index its 3 newest children on the same site. - After the crawl it reads
/ai.txt,/llms.txtand/llms-full.txtof the scanned origin, each only when robots.txt allowsAIVisibilityBotthere. Then, when it is switched on, the firewall probe below requests up to 3 of the scan's URLs again with the user agents of AI bots. - When it is switched on, JavaScript rendering below then opens up to 3 of the crawled pages in a headless browser; every request a page makes there is fetched by this crawler, under these rules.
- The homepage request follows any redirect our safety checks allow: onto a path robots.txt disallows, or even to another site without reading that site's robots.txt; only that one page is then read. Other pages, sitemaps, the well-known files and the firewall probe refuse a redirect that leaves the site or lands on a path robots.txt disallows. Pages, the well-known files and the probe also refuse one to another subdomain: they accept only the scanned host and the host the homepage redirected to (for example
www.). - If a scan's crawl is interrupted (a server restart or an internal error), it can start again within the same scan and fetch those pages, and send the firewall probe's requests, a second time.
How fast
- At most 4 requests to one host and port are in flight at a time from one server, and they start at least 250 ms apart. The limit is per host and port: http and https links to one host count separately.
- The redirects inside one request (at most 5) are followed at once, without that spacing.
- The limit and the spacing count the host a request starts at. A redirect to another host (for example from the apex to
www.) is not counted against that host, so it can get more than 4 requests at once. - One fetch, its redirects included, is stopped after 30 s; a response over 5 MiB is dropped, and so is one with a header line over 32 KiB (its line ending included), 100 header lines or more, or more than 256 KiB of headers in all.
- A page fetched in the last 24 hours, robots.txt included, is reused by a new scan instead of being fetched again.
- New scans of one site: at most 1 per UTC day from the public form, and at most 1 per UTC day that we start ourselves; a scan that fails is not counted. On top of that, a visitor's "Scan again" fetches every page fresh, at most 10 times per site per UTC day.
Where it crawls from
Requests come from these addresses: 206.81.27.2.
How to block it
Add this group to your robots.txt:
User-agent: AIVisibilityBot Disallow: /
A scan of your site then reads robots.txt and nothing else. Because a new scan may reuse a robots.txt read in the last 24 hours, a new rule can take that long to reach every scan; "Scan again" always reads it fresh. You can also block the addresses above.
Blocking AIVisibilityBot in robots.txt also stops the firewall probe below: it sends nothing to a site whose robots.txt disallows AIVisibilityBot.
Firewall probe
The probe is switched on: every scan runs it, within the rules below. When it runs, after the crawl, the check requests a few URLs of the scanned site again with the user agents of AI bots, to see whether a firewall or CDN treats AI bots differently from other visitors.
- It requests at most 3 URLs: the homepage and the next 2 crawled pages. Each is requested once with the user agent of each of these AI search and user bots the report scores: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User, Bingbot and Amazonbot. That is at most 18 requests per scan. Anthropic publishes only the names of its bots, so for Claude-SearchBot and Claude-User the probe sends their user agent as it is seen in the wild; for every other bot it sends the user agent its vendor publishes.
- A URL is requested with a bot's user agent only when robots.txt allows that path for both
AIVisibilityBotand that bot. A redirect is followed only to the scanned host or the host the homepage redirected to, and only onto a path robots.txt allows for both. - It reads at most 64 KiB of each answer; a longer answer is cut off there and its connection closed.
- It stops when the site throttles us: nothing is sent when the crawl was stopped by a 429 or 503 answer or ran out of the scan's time or data budget, and the first probe answered with a 429 or 503 (other than a bot challenge) ends the probing. The probes go out in groups of 4, so the requests already started in that group, at most 3 more, are still sent.
- These requests count towards the 4 requests in flight and the 250 ms spacing under "How fast". They are never served from or saved to the page cache, so every scan sends them anew.
- They come from the addresses above, not from the bots' owners, so your logs show one of those bots' user agents from our address. A firewall that checks a bot's address may refuse them even when it lets the real bot through.
JavaScript rendering
Rendering is switched on: every scan runs it, within the rules below. When it runs, after the crawl, the check opens up to 3 pages of the scanned site, the homepage and the next 2 crawled HTML pages, one after another in a headless browser, to compare the text a visitor sees once the page's JavaScript has run with the text in its HTML. Some AI crawlers run JavaScript and some do not.
- The browser has no network of its own. Every request a page makes in it is fetched by this crawler, from the addresses above, with the same safety checks, the same address pinning and the same limits as the crawl.
- The page itself is the copy the crawl fetched: it is not fetched a second time.
- Only GET requests for scripts, XHR and fetch calls and stylesheets are sent. Images, fonts, media, frames, workers, beacons and sockets are never loaded: the browser drops those requests and nothing is sent.
- Every request carries the user agent above, and the page sees the same user agent in the browser.
- These requests count towards the 4 requests in flight to one host under "How fast", but they do not wait for the 250 ms spacing: a page asks for its scripts and stylesheets in one burst, as it does for any visitor.
- robots.txt for
AIVisibilityBotdecides every request, on every host a page calls, other sites (a CDN, an API) included. For the scanned site it is the robots.txt the crawl read; another host's is read once per scan. There a 404, 410 or other 4xx answer counts as no rules, and a 408, 429 or 5xx answer, or a failure, keeps the scan away from that host. A request that redirects to another site is followed only when a page of the scan has already asked that site for something directly, so its robots.txt was read then: no robots.txt is read in the middle of a redirect. - Each page may send at most 60 requests and call 8 hosts, the hosts its requests redirect to included, and runs for at most 10 s; then it is read as it stands, and the requests still under way are cancelled. It is handed at most 8 MiB of answers: once it has them no new request is sent, but the answers already under way, at most 4 at once, are still read, each up to the 5 MiB limit under "How fast", and then dropped. One scan's pages may call at most 12 hosts other than the scanned site's.
- Nothing is rendered when robots.txt disallows
AIVisibilityBotfor/, or when the crawl was stopped by a 429 or 503 answer or ran out of the scan's time or data budget. A 429 or 503 answer from the scanned site stops the rendering after the current page; one from another host stops the requests to that host for the rest of the scan, and nothing else. - Rendered pages are never served from or saved to the page cache, so every scan renders them anew.
Contact
Questions or a problem with the crawler: bot@aivisibilityfree.com.