Skip to content
metryn

How to write an llms.txt file and test it

How to write a valid llms.txt, check it with curl and Lighthouse, and what the file does and doesn't do. Based on llmstxt.org and Lighthouse's source.

By Metryn · Updated 16 October 2026 · 9 min read

If you ran Lighthouse recently and saw an "Agentic Browsing" section with an llms.txt line in it, you're not imagining things. Lighthouse 13.2 added that category on 1 May 2026, and 13.3 put it in the default configuration a week later. It's experimental, and it doesn't use the usual 0 to 100 score. It reports a ratio of checks passed instead.

This guide is the practical part: how to write the file, how to test it, and what to expect afterwards. If you first want to know whether you need one at all, the short answer is in https://getmetryn.com/guides/do-you-need-llms-txt-or-special-ai-files

What the file is

llms.txt is a proposal published at llmstxt.org. You put a Markdown file at /llms.txt on your site. It gives a short description of who you are and a list of links to the pages that matter, so a language model or an agent that lands on your site doesn't have to guess the structure. It's not part of robots.txt, nothing enforces it, and no standards body owns it.

A minimal example

The proposal is strict about order and loose about everything else. An H1 with the name comes first, and it's the only thing the proposal requires. Then a short blockquote summary, then optional plain text with no headings, then H2 sections that are lists of links, each link optionally followed by a colon and a note. An H2 section called "Optional" is, by convention, the material an agent can skip when it needs something shorter.

# Northfield Outdoor Supply

> Online shop for camping and hiking gear, shipped from Leeds to the UK and EU. Prices in GBP, returns accepted within 30 days.

We sell tents, sleeping bags, stoves and clothing. Product pages list stock and delivery estimates. The help pages explain shipping and returns.

## Shop

- [All tents](https://northfield.example/collections/tents): Backpacking and family tents, with weights and pack sizes
- [Stoves and fuel](https://northfield.example/collections/stoves): Gas and multi-fuel stoves

## Help

- [Shipping and delivery](https://northfield.example/help/shipping): Costs and times per country
- [Returns policy](https://northfield.example/help/returns): 30 days, unused items

## Optional

- [About us](https://northfield.example/about)
- [Blog](https://northfield.example/blog)

Use absolute URLs and one link per page that matters, and write the notes the way you'd explain the page to a new colleague. The proposal sets no number of links. The point of the file is that someone picked the pages, so a dump of your whole sitemap defeats it. The sitemap already exists for that.

What Lighthouse actually checks

Most posts about this audit describe it loosely, so it's worth reading the code. In Lighthouse's repository, core/audits/agentic/llms-txt.js (main branch, read on 13 October 2026), the audit fetches /llms.txt from the origin of the page being tested and then does the following.

If the request fails outright, or the server answers 5xx or 429, the audit fails. If it answers any other 4xx, including a plain 404, the audit is marked not applicable, so having no file is not a failure. If it gets content, it looks for three things: a line that starts with # followed by whitespace and text, at least one Markdown-style [text](url) link, and a length of 50 characters or more. Missing any of them fails the audit, with a message saying which one.

That's all it does. In the code I read, it doesn't validate the section structure, doesn't follow your links, and doesn't look at the content type. Chrome's documentation page for the audit is shorter than the code: it says the audit flags a server error when retrieving the file, and that a missing file is not applicable because providing one is optional at the moment. If the page and the code ever disagree, trust the code, and keep in mind it can change.

How to check yours

The first test takes ten seconds.

curl -sS -D - https://yoursite.example/llms.txt | head -n 20

You want a 200 status, the file's text in the body, and no HTML. If you see your homepage or a login page, your server is answering unknown paths with a page and a 200, which is worth fixing for its own sake. Check also that the file is on the exact hostname people use: www.yoursite.example and yoursite.example are different origins, and Lighthouse looks at the origin of the page it tested.

Then read it yourself, top to bottom. Is there one H1? Does every link open? Are the descriptions still true? A file that lists pages you've deleted is worse than no file.

To see what Lighthouse says, run it from Chrome DevTools or the command line and open the Agentic Browsing category. The Chrome documentation says testing it needs Chrome 150 or later and that the category is experimental and based on proposed standards, so expect it to move.

What it does and doesn't do

Here's the plain version. Google's AI optimization guide (last updated 10 July 2026) says you don't need new machine-readable files, AI text files, markup or Markdown to appear in Google Search. On llms.txt it says it's fine to create and maintain one, that doing so will neither harm nor help your visibility or rankings in Google Search, and that Google Search ignores them. Search Engine Journal's coverage from May 2026 reports John Mueller comparing the file to the keywords meta tag. I couldn't check that against the original audio, so treat it as reported speech.

As far as the documentation shows, none of OpenAI's, Anthropic's or Perplexity's crawler pages describe llms.txt as something their crawlers read or require. Their pages are about user agents, robots.txt and IP ranges. OpenAI and Perplexity do publish an llms.txt for their own documentation, which says something about how they like to present docs to agents and nothing about how they treat yours. I haven't found any provider that says the file changes what appears in its answers.

What Chrome's audit says it is for is narrower: a summary an agent can use once it's already on your site. That's a modest claim, and a reasonable one. Whether a given tool reads the file is something you can only learn from your own server logs.

So the realistic reasons to publish one are that it takes an hour, it's easy to keep accurate, and a Lighthouse line goes green. Rankings, citations and traffic aren't on that list, because no operator documents them and we won't invent a number.

A caveat that matters more than the file

A file is only read if the request gets through. Hosts and CDNs often have bot-management or firewall rules that block or challenge automated traffic by user agent or category, and a rule written for scrapers can also catch an AI search crawler. Perplexity's crawler page has Cloudflare and AWS WAF sections for exactly this reason, and it recommends matching the user agent together with the published IP ranges.

It's a reason to look once, not to worry. Open your robots.txt and see what it says for OAI-SearchBot, Claude-SearchBot and PerplexityBot. Then look at your firewall or bot-management settings for rules that match those user agents or their categories. OpenAI, Anthropic and Perplexity all publish the IP ranges their crawlers use. A check from outside, like the free scan below, shows what a crawler-like request gets from your site. It runs from our servers, not from the operators' networks, so treat it as a pointer rather than proof.

Questions people ask

Does a missing llms.txt fail the Lighthouse audit?

No. In the audit's source, a 404 or any other 4xx answer other than 429 is marked not applicable. Chrome's documentation gives the same reason: providing the file is optional at the moment. A 5xx, a 429 or a failed request does fail it.

Do I need an llms-full.txt as well?

The Lighthouse audit I read only requests /llms.txt. The llmstxt.org proposal is the place to look if you want to know what else it describes. Nothing in Google's, OpenAI's or Perplexity's documentation that I read asks for a second file.

Will Google use it?

Google's AI optimization guide says Google Search ignores llms.txt and similar files, and that creating them neither harms nor helps your visibility or rankings there. It's fine to publish one for other reasons.

Should it list every page?

No. The proposal describes a curated list with short notes, and your sitemap already covers the complete list. If a page isn't one you'd point a new colleague to, leave it out.

Does it belong in robots.txt?

No, they're separate files. robots.txt says what a crawler may request. llms.txt is a summary for something that has already arrived. Blocking a crawler in robots.txt or in a firewall is not undone by a friendly llms.txt.

What to do this week

  • Run the curl command from this guide against both the www and the bare hostname.
  • Check that the response is a 200 with plain text or Markdown, not your homepage.
  • Read the file and remove every link to a page that no longer exists.
  • Open your robots.txt and your firewall rules and look for anything that matches AI search crawlers.
  • Put a reminder to re-read the Lighthouse audit's documentation in a few months, because it's experimental.

Check your own site

Paste your address below. The free scan looks at robots.txt, your sitemap, access for the main AI crawlers and structured data, and shows what it found in seconds. It tells you what a machine can fetch and read. It doesn't predict rankings or citations.

Free scan · No account needed · Results in seconds

Sources

Keep reading

Want to check one file yourself? Try the free AI crawler robots.txt checker or the llms.txt validator, or read how the score is built.