Skip to content
BlogRainbow AI Ready

Troubleshooting crawls

Common reasons a crawl fails or a page is missing.

"robots.txt returned a server error"

Crawlers must treat an unreachable robots.txt (HTTP 5xx) as "disallow everything". Fix your robots.txt and start a new crawl.

A page is missing from llms.txt

Check the crawl log on the Crawl history tab. Common reasons:

  • disallowed by robots.txt, or marked noindex
  • matched a default exclusion (cart, account, search, parameter URL…) or one of your exclude patterns
  • beyond your plan's page limit or the maximum depth
  • the page needs JavaScript (enable JavaScript rendering)

Pages come out empty

Your site probably renders content with JavaScript. Enable JavaScript rendering in Crawl settings (Starter plan).

Verification fails

  • DNS: wait up to an hour after adding the record and check the record name.
  • File: make sure it is served as plain text at the exact path shown, without redirects to a login page.
  • Meta tag: it must be on the homepage inside <head>.

The badge shows "Needs attention"

Open the website's Overview: the reasons are listed there. Run a new crawl or publication check after fixing them.