AI Crawlability Checker
Enter your domain to check the crawler rules in your robots.txt file and see which AI bots are blocked.
Trusted by brands
Trusted by agencies
A single robots.txt line can keep you out of AI answers
Crawler rules are edited by whoever ships the site, rarely reviewed, and never reported back. You only notice when a competitor is quoted and you are not.
One line written for scrapers closes the door on AI search
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /Search indexing and model training are separate decisions
Feeds the answers your buyers read today. Blocking it removes you from that surface.
Feeds future model versions. Keeping it closed is a legitimate policy choice.
The check reports the two groups separately, so a deliberate training policy is never scored as a defect.
An allowed page still says nothing when the copy needs JavaScript
<div id="root"></div>
<script src="/app.js"></script>The check flags server-rendered content, JavaScript dependence, and how much of the page is exposed without it.
Fixed the crawler rules? Find out what AI says about you next
Access is the floor, not the outcome. Peekaboo runs your buyer prompts through the real AI interfaces and shows whether your brand gets named.
2 minutes. No credit card.
Three passes, one honest verdict
The check reads the same file the crawlers read, applies the rules the way they do, and tells you exactly which line decided each answer.
See every AI crawler, grouped by its job
Nine named crawlers are evaluated against your file, split into search indexing, model training, and user-triggered fetching, so one policy never hides another.
- Search indexing crawlers reported apart from training crawlers
- User-triggered fetchers marked where robots.txt is advisory
- Wildcard and named user-agent groups resolved separately
Read the line that made the decision
Your robots.txt comes back in full, so you can see the wildcard group, the named group, and the exact directive that won each match.
- The most specific user-agent group wins, as crawlers apply it
- The longest matching path wins, and an allow wins an exact tie
- Private paths you already disallow are carried through, not erased
User-agent: *
Allow: /
Disallow: /api/
Disallow: /studio/
Sitemap: https://www.example.com/sitemap.xmlTake a reviewable block back to your repo
The generated block opens search crawlers, preserves the training choice you already made, and keeps your private paths closed. It is written to be merged, not to replace your file.
- Copy it, review it, then merge it into the robots.txt you own
- Your unrelated search-engine rules are never rewritten for you
- Allowing a crawler does not guarantee a citation or a mention
# AI crawler policy block generated by Peekaboo
# REVIEW BEFORE PUBLISHING.
User-agent: OAI-SearchBot
Allow: /
Disallow: /api/
User-agent: GPTBot
Allow: /
Disallow: /api/“It’s been very helpful for understanding exactly how the brands we manage are showing up in LLMs and answer engines. The precise prompt tracking, along with the broader set of related prompts, is extremely valuable for showing us what to take action on and where content gaps exist.”
Ash RomoSenior Manager, Brand Reputation and Social MediaAgent AdviceWe track what AI says
Ongoing tracking records answers to your selected prompts through supported AI interfaces. Each free tool checks a specific question, with its scope explained above.
For most online stores, three names come up consistently: Reviews.io, Trustpilot and Judge.me. Reviews.io stands out for verified buyer reviews and
Captured from actual interfaces
Processed in real time
Track brand performance
Copilot · Coming soon
Track five supported AI search surfaces. Claude and Microsoft Copilot tracking are coming soon.
Book a demo“I really appreciate the team at AI Peekaboo. We have their contact details on Slack, and they're always available to jump on a call. The usability of the platform is a big plus for me. The dashboard is super easy to use, and the UX/UI is really user-friendly. The information provided in terms of AI insights and competitor analysis is going to be crucial in the agentic commerce era. AI Peekaboo automates crawling through different LLMs, which used to be a manual task for us. The dashboard setup makes it easy to rank competitors versus us, and it's helpful to have direct contact with the team to answer questions and get support. Setup was super easy for my team member, with great onboarding support, including weekly calls with the AI Peekaboo team.”
Ellie BradfordContent ManagerWhat an allowed crawler does and does not buy you
Three limits worth stating plainly before you take the result to a stakeholder.
Access is permission, not a promise
Removing a disallow lets a crawler request your pages. It never guarantees crawling, indexing, or a mention.
Blocking training stays your choice
These rules state your preference for the named training crawlers. They do not remove previously collected content. The check does not score that choice.
Some AI fetchers never read the file
User-triggered fetchers run when a person asks for your page, and providers say robots.txt may not apply.
Hear what leading marketers say about Peekaboo
Peekaboo became an essential part of our marketing intelligence stack. It gives us clear visibility into where and how our brand is mentioned across LLMs, and helps us identify the content being picked up and how we compare to competitors.
We knew LLMs were driving traffic, but we had no visibility into where we appeared. Peekaboo gave us clear insight into our mentions, our gaps, and opportunities. It is now essential for tracking and improving our share of voice across AI platforms.
The team behind Peekaboo has been great to work with. They're quick to help, open to feedback, and genuinely care about improving the product. It feels like they're partners in our success, not just software providers
Questions, answered
Is it really free?
Yes. No login, no email gate. Drop a domain, get the check.
How often should I re-check?
Whenever you change robots.txt, deploy a new framework, or add a CDN. Crawler rules are easy to break by accident.
What does allowing an AI crawler change?
It removes a robots.txt directive that prohibited the named crawler from accessing the affected paths. It does not guarantee crawling, inclusion, citation, a mention, or a ranking change.
