llms.txt Builder

About llms.txt Builder

llms.txt Builder discovers pages on a public site, lets you tick a curated set, then writes a markdown index with one-line descriptions. Enter a domain or any URL, click Find my pages. Nothing is auto-ticked. Generating a file does not upload it to your origin.

Discovery reads robots.txt, then sitemap.xml (including a sitemap index one level down and .xml.gz). If that fails, it crawls the homepage two levels deep, capped at 200 URLs. example.com returned a paste box (pasteNeeded) because there was no sitemap and no followable homepage list. Use pasted URLs or start from SaaS product, Documentation site, or Content site.

Suggested selection ticks indexes, not every post. Create llms.txt then fetches a title and description for each selected URL (select cap 2,000). Sections group by top-level path. Ten links show; View all expands. Click a title or description to edit. This is not a sitemap replacement.

When to use it

Use it when you want a short map for agents that choose to read /llms.txt, after the important URLs are stable. Use it when a competitor is cited from a handful of canonical pages and your crawl file is a 2,000-URL dump. Rebuild after a nav change, not after every blog post.

How to use llms.txt Builder

1. 1Enter the site and click Find my pages. A domain or any URL on the origin. Step 1 reports a sitemap list, a homepage crawl, or a paste box.
2. 2Tick the pages that belong. Open What belongs in an llms.txt if you want the rule of thumb. Suggested selection ticks indexes. Select all is per group. Groups start collapsed.
3. 3Click Create llms.txt. Selected pages are fetched for title and description. Edit those lines on the cards. View all expands a section past the first ten links.
4. 4Copy or download the markdown. Save it at the site root as /llms.txt. Use a template if you have no live site yet. Keep sitemap.xml for Google.

Why this matters for AI search

A curated summary beats a full index for a reader with a context limit. An agent that opens llms.txt will not ingest your entire sitemap. It will take the first useful cluster of links and descriptions. A 200-URL homepage crawl that you tick in full is a directory, not a brief. Suggested selection exists so the first pass is 10 to 40 index pages.

That is why nothing starts ticked, and why a pick over 40 warns. The file is for a model that has a window, not for Googlebot. Search Central says AI Overviews and AI Mode do not need special AI text files. Keep sitemap.xml for search. Put llms.txt at the site root if you want agents that look for it to see a short, edited map.

What this tool cannot do

It cannot prove any model opened the file, execute JavaScript while discovering links, or replace sitemap.xml. Five discovery runs per hour. Localhost is blocked. Titles after Create llms.txt come from the first HTML response of each selected URL. Google Search does not read llms.txt.

Common mistakes

Ticking every discovered URL. A full dump wastes the window the file is meant to save.

Treating Find my pages as a publish step. You still have to save the file at /llms.txt yourself.

Ignoring the paste box on a JavaScript homepage. Discovery will not invent routes the HTML does not link.

Deleting sitemap.xml because you now have llms.txt. Google Search does not use this file.