Can the robots get in the door?
AI assistants send small programs, called bots, to visit websites and read them. Some sites lock these bots out without the owner knowing, often through a security setting.
What we do: we read the rules your robots.txt file (the short file that tells bots where they may go) sets for 6 AI bots, then visit your site as each of them and see which get in.
For your developer
We read robots.txt for the rules on OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User, Bingbot and Amazonbot, for your homepage and every page we crawl, then request up to three of your pages with each bot's published user agent to test your firewall and CDN. We also look for a Content-Signal opt-out in robots.txt and for an /ai.txt that blocks every bot.
Our requests come from our server, not from the bots' own addresses, so a firewall may treat the real bot differently.