About our crawler

The AI visibility check fetches a website when someone asks for a report on it. We also prepare reports on businesses we intend to contact: those scans are started from our own list by a timer, at most 1 per site per UTC day. The crawler has no other schedule.

How it identifies itself

User agent: AIVisibilityBot/1.0 (+https://aivisibilityfree.com/bot)

Robots token: AIVisibilityBot. The token is permanent: if the user agent is ever renamed, rules written for AIVisibilityBot keep applying.

What it fetches

How fast

Where it crawls from

Requests come from these addresses: 206.81.27.2.

How to block it

Add this group to your robots.txt:

User-agent: AIVisibilityBot
Disallow: /

A scan of your site then reads robots.txt and nothing else. Because a new scan may reuse a robots.txt read in the last 24 hours, a new rule can take that long to reach every scan; "Scan again" always reads it fresh. You can also block the addresses above.

Blocking AIVisibilityBot in robots.txt also stops the firewall probe below: it sends nothing to a site whose robots.txt disallows AIVisibilityBot.

Firewall probe

The probe is switched on: every scan runs it, within the rules below. When it runs, after the crawl, the check requests a few URLs of the scanned site again with the user agents of AI bots, to see whether a firewall or CDN treats AI bots differently from other visitors.

JavaScript rendering

Rendering is switched on: every scan runs it, within the rules below. When it runs, after the crawl, the check opens up to 3 pages of the scanned site, the homepage and the next 2 crawled HTML pages, one after another in a headless browser, to compare the text a visitor sees once the page's JavaScript has run with the text in its HTML. Some AI crawlers run JavaScript and some do not.

Contact

Questions or a problem with the crawler: bot@aivisibilityfree.com.

Back to the AI visibility check ยท Privacy