AI Crawlers

AI assistants send bots to read web pages. AI Crawlers shows which bots read your site, which pages they read, and which of those pages your tracked prompts cite. The data comes from real requests to your site, not from prompts.

AI Crawlers appears in the sidebar under Traffic when it is turned on for your workspace. Without an active plan, the page shows a plan notice instead of data. Don't see it? Ask us at team@aipeekaboo.com.

What the page shows

• Reads by AI bots: a chart of reads over time. A read means an AI bot loaded the page.
• Company cards: one card each for ChatGPT, Perplexity, Claude and Google, with reads and citations. Google's AI reads through Googlebot, so its card shows Reads not measured. Every other AI bot, including Google's and Microsoft's AI agents, is grouped under Other bots.
• Pages: the pages bots read or that your tracked answers cite. The columns are Reads, Citations and Cited in. Which ones show depends on the group you pick and your screen width.

Pages are split into these groups:

• Read, not cited (shown first)
• Read and cited
• Cited, not read
• Errors
• All

Reads and citations

A read is a bot loading your page. A citation is one of your tracked answers in Prompts naming that page as a source. Citations counts those answers, and Cited in lists the AI models. A page can be read often and never cited, or cited with no read we could see. See Visibility vs citations and Sources and citations.

Which hosts can connect

Your setup screen lists the hosts open to your workspace.

• Cloudflare: you add a small Worker to your site in Cloudflare. Your site must run through Cloudflare (proxied).
• Shopify: a Shopify theme store whose domain uses Cloudflare. Shopify says it doesn't support this setup and that it could break at any time. The steps are in Connect a Shopify store to AI Crawlers. Read it before you decide.

For headless Shopify (Hydrogen) or another host such as Fastly or CloudFront, ask us at team@aipeekaboo.com. Site builders such as Wix can't be connected.

How to start

1. Open AI Crawlers in the sidebar and pick your brand in the header. If the page says Set your site URL first, add your site address in Settings.
2. If the page asks how your site is served, pick your host. Read the cost and requirements it shows.
3. On Cloudflare, choose Create key and start setup. The page makes your site key and gives it to you inside your own worker.js. On Shopify, follow the Shopify steps; the key is made at step 4.
4. Follow the steps on the page. Most are done in your Cloudflare account, and the page shows each value to copy.

The setup is done in Cloudflare. If someone else manages your Cloudflare account, do the steps in PeekABoo together with them, or pass worker.js over a private channel such as a password manager.

Copy the key right away: we can't show it again.

If you lose it, choose New key on the page, then paste the new worker.js into your Worker and choose Deploy. Bot requests before you deploy aren't recorded. A new key takes about a minute to start working, and the old key stops working within a minute.

Check that it works

Open the test link. It should say sent. Then choose Check now. The chip changes to Test received.

The test is not a bot visit, so it does not open your data. The page opens by itself when your first bot requests arrive, or after we review your first data, and we can't say when. Once it is open, it shows "Connected. Waiting for the first AI bot." until an AI bot reads a page. Bots usually arrive within a few hours to a few days.

What is sent

The Worker sends only these:

• Only requests whose name looks like a bot's
• Address, method, status, time, bot name, request id, linking site
• No visitor IPs or cookies

The address is your site's host plus the page path, without the query string. The bot name is the bot's user-agent text, cut at 1,024 characters. The linking site is the address of the site that linked to yours, with no path. The test link sends one test record. Keep any path that can carry something private, such as an order status or account page, off the Worker's route.

What it costs

On Cloudflare's Free plan, all Workers on your Cloudflare account together get 100,000 requests a day, and every request to the route counts, not only bot requests. Past that limit, with the route set to Fail open (proceed), your site is served as usual and no data is recorded until the limit resets at midnight UTC. With Fail closed (block), visitors can see an error page. Above the free limit you need Cloudflare's paid Workers plan, from $5 per month.

What it cannot see

• Reads by Google's AI. They come through Googlebot, so Google shows Reads not measured. Reads by Microsoft's AI come through Bingbot and are not measured either.
• Bots that Cloudflare blocks before the Worker runs. They look the same as bots that never came.
• Bots whose name doesn't look like a bot's. Only bot-like requests are kept.
• Visitors. Requests that don't look like bots are not recorded.
• Reads from before you connected. The page starts empty.
• Shopify checkout. Cloudflare doesn't run Workers there.
• What an assistant told someone. A read shows that a bot loaded a page. It doesn't show that an answer used it.

Nothing arriving?

Check that the route covers your address, that you pasted your worker.js and deployed it, and that the test link says sent. After making a new key, wait about a minute.

Still nothing after a few days? Write to team@aipeekaboo.com or see Contact support.