Troubleshooting crawls
Common reasons a crawl fails or a page is missing.
"robots.txt returned a server error"
Crawlers must treat an unreachable robots.txt (HTTP 5xx) as "disallow everything". Fix your robots.txt and start a new crawl.
A page is missing from llms.txt
Check the crawl log on the Crawl history tab. Common reasons:
- disallowed by robots.txt, or marked
noindex - matched a default exclusion (cart, account, search, parameter URL…) or one of your exclude patterns
- beyond your plan's page limit or the maximum depth
- the page needs JavaScript (enable JavaScript rendering)
Pages come out empty
Your site probably renders content with JavaScript. Enable JavaScript rendering in Crawl settings (Starter plan).
Verification fails
- DNS: wait up to an hour after adding the record and check the record name.
- File: make sure it is served as plain text at the exact path shown, without redirects to a login page.
- Meta tag: it must be on the homepage inside
<head>.
The badge shows "Needs attention"
Open the website's Overview: the reasons are listed there. Run a new crawl or publication check after fixing them.