How it works
From your URL to a verified, published llms.txt
A careful pipeline that respects your site and never invents facts.
- Step 1
Add your website
Enter your URL. We check robots.txt, find your sitemaps, detect your platform and produce a first readiness report.
- Step 2
Verify ownership
Add a DNS TXT record, upload a small file, or paste a meta tag. You can preview your files before verifying.
- Step 3
We crawl and generate
Allowed pages are fetched politely, converted to clean Markdown, and turned into llms.txt, llms-full.txt and JSON files.
- Step 4
Publish and stay current
Files go live on your hosted URL (or your domain via the plugin) and refresh on your schedule.
Inside each crawl
- 1.DiscoverWe read robots.txt and your sitemaps, then rank URLs so the pages that matter most (home, about, products, services, docs, policies) come first.
- 2.Fetch politelyOne page at a time, honouring your Crawl-delay. Unchanged pages are recognised with HTTP caching headers and skipped.
- 3.ExtractEach page becomes clean Markdown. Structured data gives us products, prices, FAQs and contact details exactly as published.
- 4.DescribeA local AI model writes short page descriptions. A fact guard rejects any sentence with numbers, prices, emails or links not found on the page.
- 5.Generate and validatellms.txt, llms-full.txt, Markdown and JSON files are built and validated: structure, links, encoding, sizes and schemas.
- 6.Publish and verifyValid files are published atomically as a new version. We then fetch the public URL to confirm it serves exactly what we published.
More details: crawl settings, publishing options, and how we keep AI text factual.