For most of the web's life you had two audiences to write for: people, and the search crawlers working on their behalf. Googlebot and Bingbot read your HTML, followed your sitemap.xml, and obeyed robots.txt. That arrangement held for twenty-five years.
Then people started asking ChatGPT, Claude, and Gemini to summarise a page instead of clicking through to it. The reader changed. And the way that new reader consumes your site is nothing like a browser tab.
When a model or an agent pulls your page, it does not see the clean article you see. It sees the raw HTML underneath: framework bundles, tracking pixels, nav menus, a cookie banner, a couple of ad slots. To a language model all of that is noise. It eats context window, it costs tokens, and it buries the one paragraph that actually answers the question. Bury that paragraph deep enough and the model starts to guess. Guessing is where wrong answers about your pricing come from.
llms.txt is the community's answer to that problem. If you want to know whether your own site already publishes one, the llms.txt check will tell you in a couple of seconds. The rest of this piece is about what belongs in the file and why it helps.
What llms.txt actually is
It is a single Markdown file living at the root of your domain, at https://yourdomain.com/llms.txt. Think of it as a curated index written for machines: a short statement of what your site is, followed by a hand-picked list of the pages that matter, each one a plain Markdown link.
It sits next to two files you already know, but it does a different job from either of them.
| File | Written for | Its job |
|---|---|---|
robots.txt |
Web crawlers | Access rules: where bots may and may not go |
sitemap.xml |
Search indexers | A complete inventory of every URL |
llms.txt |
AI models and agents | A short, curated Markdown guide to what matters most |
The contrast with sitemap.xml is the clearest one. A sitemap says "here is everything." An llms.txt says "here is what to read first, and here is what each thing is." One is exhaustive; the other is opinionated. That editorial judgement is the entire point of the file.
What goes inside the file
The format is deliberately plain, which is part of why it works. A top-level heading names the site. A blockquote gives a sentence or two of context. Then ## section headings group your links, and each link is an ordinary Markdown bullet with a short description after it.
# Example Corp API & Platform Documentation
> Example Corp provides high-performance payment processing infrastructure and global financial APIs.
## Core Documentation
- [API Overview](https://example.com/docs/api): Complete REST API reference.
- [Authentication](https://example.com/docs/auth): OAuth2 and API key guides.
## Product Guides
- [Pricing](https://example.com/pricing): Enterprise pricing tiers.
That is the whole shape: an # title, a > summary, ## sections, and - [Title](URL): description links. Some sites also publish a companion /llms-full.txt that concatenates the actual documentation into one plain-text file, handy for retrieval systems that want the text in a single request. It is optional, and you can add it later.
Habits worth keeping, even though the check won't grade them
A few things separate an llms.txt that helps from one that just exists. None of these are pass-or-fail rules, and the ServerRecords tool does not test any of them, but they are the difference in practice.
- Link health. The links you list should still resolve. A curated index pointing at three
404s is worse than no index, because you have told the model exactly where to find pages that are gone. - Token efficiency. The reason the file exists is to save the model from wading through HTML. Keep it lean. A tight list of the pages that matter beats a dump of every URL you own.
- Content type. Serving the file as
text/plainortext/markdownis the well-behaved choice; a file mislabelled astext/htmlinvites the wrong parser. - Bot access. If a permissions layer or a strict CORS policy blocks the agents you are trying to help, the file might as well not be there. Make sure it is reachable.
Treat these as the background you are aiming at, not a report card. They matter for how well the file works once a real model reads it.
What the ServerRecords check looks at
Here is the honest scope, so you know what a green result does and does not mean. The llms.txt check fetches the /llms.txt path on your domain and confirms three things: that the file is actually there (an HTTP 200), that it opens with a top-level Markdown heading (a line starting with # ), and how many Markdown link lines it contains. That is the extent of it.
In plainer terms, the tool confirms the file is present and has a basic, well-formed structure: a heading and a list of links. It is a fast sanity check for "does this exist and is it shaped like an llms.txt at all," which is exactly the question you want answered first.
What it deliberately does not do is worth stating. It does not follow the links to check they resolve, so link health is on you. It does not count tokens or score how well you have curated the file. And it does not inspect the content type, CORS headers, or bot accessibility. Those are the good habits from the section above, and they stay your responsibility.
One related check sits nearby. Because everything here assumes the file is served over a working HTTPS setup, the HTTP/HTTPS check is the tool to reach for if you want to confirm the domain itself responds cleanly and redirects plain HTTP where it should.
Where to start
If you have never published one, start small: a title, a one-line summary, and links to your five or six most important pages. Put it at /llms.txt, run the check to confirm it is live and structured, then click through your own links once to be sure none of them 404. You can grow it from there. As more of your readers arrive by way of a model rather than a search result, a clean, honest llms.txt is a cheap thing to get right early.