llms.txt: the new robots.txt — what it is and whether you need one

llms.txt is a plain-text file in your domain root (yoursite.com/llms.txt) that summarizes, in plain language with markdown headings and links, what your site is, which pages matter and where information lives. Where robots.txt tells a bot what it may fetch, llms.txt tells a language model where to look. It's not a law, not a W3C standard, and not a guarantee of a citation — but it's a cheap way to save a model the work of combing through your entire site.

Do you need one? On a small, tidy site with a handful of pages and solid schema.org markup, it adds little — the structure already speaks for itself. Once your site grows to dozens of pages, multiple products or showcases (like this one), llms.txt becomes the shortest route to a model that grasps what you offer at a glance, instead of having to crawl ten pages itself to figure it out.

Where does llms.txt come from?

The proposal comes from Answer.AI (Jeremy Howard) and surfaced in 2024 as a response to a concrete problem: language models work with a limited context window, and a full website — navigation, footers, cookie banners, markup — is an expensive, noisy way for a model to reach the substance. The idea: layer a clean, human-and-machine-readable index on top, in the same plain-text format a model already handles well.

It deliberately borrows its name and location from robots.txt — the place bots have checked first for 30 years — but the purpose differs. robots.txt is a permission file: crawl this, not that. llms.txt is a navigation file: this is where the content that matters lives. They work as complements, not replacements for each other.

What does an llms.txt look like?

The proposed structure is simple markdown: an H1 with the site name, a short blockquote summarizing what the organization does, and below it H2 sections with linked page names and a one-line explanation per link. Some sites add a fuller variant at /llms-full.txt, with the full markdown content of the most important pages pasted in — so a model doesn't even need to click through.

No JSON, no schema, no required fields — that's intentional. The file should be as easy to write by hand as a robots.txt, and just as easy to maintain when a page disappears or gets renamed.

Does it actually work?

Honest answer: it's not yet an official standard, and not every AI crawler fetches the file or actively uses it when answering questions. It's not a replacement for the things already proven to work — indexable pages, a clear heading structure, schema.org markup, and a robots.txt that grants AI crawlers access. Treat it as an extra, cheap layer on top of that foundation, not the foundation itself.

Why bother, then? Because the cost is negligible — one static file, a few lines of markdown — while the odds that it helps some model somewhere only grow as more AI systems start actively consulting it. We keep the file on neuralex.nl up to date alongside our robots.txt, schema.org markup and agent cards; how those other three technical layers fit together is laid out in our piece on what GEO actually is.

Building your own llms.txt: the short version

Start with the question a model asks itself when it lands on your domain: what does this organization do, and which page answers which question? Translate that into an H1 with your organization name, one two-sentence blockquote summarizing what you do, and per section of your site an H2 with a short list of links — each with half a sentence of explanation instead of a bare URL. Leave out old or thin pages; the file should be a summary, not a sitemap.

Maintain it the way you maintain a sitemap: add a line for every new page worth mentioning, remove the link when a page is deleted or renamed. A stale llms.txt pointing at dead links is worse than no llms.txt at all — it undermines exactly the trust you were trying to build.

Conclusion

llms.txt is not a miracle fix and not a ranking factor — it's a small, cheap navigation file that saves a language model from having to search your entire site. For a site of any real size, it's a quarter-hour of work with little downside and a real chance of upside. How it fits into the broader approach — from crawler access to citable paragraphs — is covered in our pillar article on GEO.