Troubleshooting and account
Fix the pages AI bots cannot load
Find the pages AI bots fail to load on AI Crawlers, see which bots and status codes are involved, fix the usual causes and confirm the fix.
5 minute read · Last reviewed: November 17, 2026
When an AI bot asks for one of your pages and gets an error, it has nothing to read, so that page can't feed an AI answer. AI Crawlers shows which pages fail, which bots hit them and what your site answered. Most failures come from moved pages, server trouble or security rules, and each has a plain fix.
You need a site connected to AI Crawlers first.
Find the failing pages
- Open AI Crawlers and set the range in the header. Days are in UTC.
- Look at the yellow issues strip under the chart, such as 2 pages AI bots could not read. Choose View, then View pages to open the Errors group of Pages AI reads.
- In Errors and status codes, read the headline, such as 8.7% failed, with the number of failed requests out of all AI requests. Choose Failed only to see only the error bars, day by day.
- Open Pages that failed at the bottom of that card. Each row shows the Page, Why it failed (for example Not found (404) with a count) and Failed, as failed requests out of all requests for that page.
- In AI bots that visited, sort by Failed to see which bots fail most. Open a bot's row to see the pages it requested most, with how many failed.

To follow one bot, pick it in All bots above the chart. The whole page then shows that bot's requests, and its pages move to a card of their own.
What each outcome means
Errors and status codes sorts every AI request into one of these outcomes:
| Outcome | What it means | Counts as failed? |
|---|---|---|
| Loaded | Your site answered with the page or a redirect | No |
| Status not visible | Your host handled the request, so we can't see the final status | No, it counts as a read |
| Blocked | Your firewall denied, challenged or rate-limited the bot. Shown only where your connection can see firewall decisions | Yes |
| Not found (404) | The address doesn't exist on your site | Yes |
| Other 4xx | Any other client error, such as forbidden, login required or too many requests | Yes |
| Server error (5xx) | Your server failed while answering | Yes |
When your connection can't see blocked requests, the footnote at the bottom of the page says so. On Cloudflare, for example, bots that Cloudflare blocks first never reach the Worker.
Fix the common causes
Not found (404)
The page was moved or deleted, or a link points to an address that never existed.
- If the page moved, add a permanent (301) redirect from the old address to the new one.
- Fix internal links and your sitemap so they point to live pages.
- If the address never existed and nothing links to it, a bot may just be probing. You can leave it.
A bot that only got errors shows under Came but read nothing, so not in the chart, with its request count.
Other 4xx
Usually a security rule or a login wall turned the bot away.
- Check your host's or firewall's bot settings and allow the AI bots you want to read your site.
- Keep public content outside login pages.
- If bots get "too many requests", raise the rate limit for verified AI bots.
Server error (5xx)
Your server failed while the bot waited. Check your host's error logs for the failing page, and ask your developer or host. Errors that cluster on one day usually point to an outage or a deploy.
Blocked
When a bot is blocked on several pages, the issues strip shows a line such as a bot name followed by blocked on 3 pages. Choose Copy fix and send the text to whoever manages your firewall. It lists the pages and the change to make: allow requests from that company's published IP ranges, not just its bot name, because a bot name can be faked.
Blocking AI bots can be your choice. Only unblock the bots you want reading your site.
robots.txt and bots that stopped
The issues strip also warns when robots.txt is failing for AI bots, or when a company's bot stopped or its reads dropped. Check robots.txt opens the free AI crawlability checker, so you can see which AI bots your robots.txt allows.
When failures are fine
- Training bots: pick Training only in the purpose filter to see them apart. Failures there don't affect live answers.
- Probing: bots sometimes request addresses that never existed. A few 404s on odd paths are normal.
- A small share: a page with one failure in hundreds of requests doesn't need a fix.
Confirm the fix
- Open the fixed page in a private browser window and check that it loads, or that the old address redirects.
- Wait a day or two for bots to come back. The chart only changes when they do.
- In Errors and status codes, choose Failed only and check that the error bars stop from the day of the fix. Today's numbers are still coming in.
- Open Pages that failed again. A fixed page drops off once its failed requests fall out of the range, so a 7-day range clears fastest.
Still failing after a fix you are sure of? Write to team@aipeekaboo.com with the page address and the outcome shown, or see Contact support.
Frequently asked questions
Does Status not visible mean the page failed?
No. Your host handled the request, so we can't see the final status. It counts as a read, not as a failure.
Why don't I see a Blocked outcome?
Blocked shows only where your connection can see firewall decisions. On Cloudflare, bots that Cloudflare blocks first never reach the Worker, and the page footnote says blocked requests can't be seen.
How soon does a fixed page drop off the list?
Once bots read it again and its failed requests fall out of the selected range. A 7-day range clears fastest.
