Skip to content
metryn

Do you need llms.txt or special AI files?

Do you need llms.txt or other special files for AI search? What Google, OpenAI and Perplexity document, what they don't, and where your time is better spent.

By Metryn · Updated 16 October 2026 · 6 min read

Every few months a new file or tag is pitched as the thing AI search needs from your site. It's reasonable to ask which of them the operators themselves ask for.

What Google says

For AI Overviews and AI Mode, Google's documentation states that a page must be indexed and eligible to appear in Search with a snippet, and that there are no additional technical requirements. It says there are no additional requirements or special optimizations necessary, and that site owners don't need new machine-readable files, AI text files or special schema.org markup.

What OpenAI and Perplexity document

Their documentation is about crawlers and robots.txt. OpenAI tells you to allow OAI-SearchBot in robots.txt, and allow its published IP ranges, to appear in ChatGPT search. Perplexity recommends allowing PerplexityBot and permitting its published IP ranges. Neither page describes an llms.txt requirement.

So is llms.txt useless?

Not necessarily harmful, and some tools and agents may read it. But the operators above don't state it as a requirement, and nothing in their documentation says having one changes whether you show up. Treat it as an optional, low-cost courtesy at best. Metryn publishes one on its own site, labelled as an experiment, for exactly that reason.

Update, 13 October 2026: Chrome's Lighthouse now has an experimental "Agentic Browsing" category that includes an llms.txt audit (added in 13.2, in the default configuration since 13.3). Read from Lighthouse's source, the audit does a narrow thing. It requests /llms.txt from the page's origin and fails on a network error, a 5xx or a 429. Any other 4xx, a plain 404 included, is reported as not applicable, so having no file is not a failure. If a file comes back, it checks for an H1 line, at least one Markdown link and a length of 50 characters or more. The category has no 0 to 100 score, only a ratio of checks passed. What it doesn't do: it doesn't show that any AI system reads the file, and Google's AI optimization guide says Google Search ignores these files and that creating one will neither harm nor help your visibility there. The step-by-step version, with an example and a curl check, is in https://getmetryn.com/guides/how-to-write-an-llms-txt-file

Controls you do have

Google lists the levers that actually exist. nosnippet, data-nosnippet and max-snippet limit what's shown from a page in Search, noindex blocks indexing, and robots.txt directives for Googlebot control crawling. Google-Extended is the control for limiting use of your content for AI training and grounding in some other Google systems. Those are real, documented switches, and they're a more honest answer to the question of control than any single text file.

The training question and the visibility question are separate on OpenAI's side too. Disallowing GPTBot signals you don't want content used for training. It doesn't, on its own, take you out of ChatGPT search, which is governed by OAI-SearchBot. Decide each on its own merits.

Where the effort pays off instead

  • robots.txt that allows the search crawlers you want, on every hostname.
  • No firewall rule quietly blocking those crawlers' requests.
  • Important content present in the HTML response, as text.
  • Internal links that are real links, and a sitemap that lists canonical URLs.
  • Structured data that matches what the page says.

How to think about a new file or tag someone recommends

When a new file or tag gets recommended, run three quick questions past it. First, does the operator whose product it targets document it? If OpenAI, Google or Perplexity don't mention it anywhere, you're being asked to take it on faith. Second, what does it cost to add and to maintain? A file that's easy to generate and can't hurt is different from a rewrite. Third, what are you giving up by doing it instead of something else?

Applied to the fundamentals, those questions all come out the same way: robots.txt rules, content in the HTML and working links are documented by every operator, cheap to check and the prerequisite for everything else. A file nobody documents can wait until the basics are done.

Questions people ask

Will adding llms.txt hurt?

The documentation I read doesn't say it harms anything. It also doesn't say it's required, or that it changes whether you appear. It's a low-cost, optional file.

What does Lighthouse's llms.txt audit do?

As of 13 October 2026: it's part of an experimental Agentic Browsing category with no 0 to 100 score. It fetches /llms.txt from the page's origin and fails on a network error, a 5xx or a 429. A 404 or any other 4xx is reported as not applicable. If the file exists it needs an H1 line, a Markdown link and at least 50 characters. Google's AI optimization guide says Google Search ignores these files. How to write and test one is in https://getmetryn.com/guides/how-to-write-an-llms-txt-file

What does Google use for AI Overviews?

Google's documentation says the same Search systems apply: a page has to be indexed and eligible to show a snippet, and there are no additional technical requirements for AI features.

What's the point of controls like Google-Extended?

Google describes it as the control for limiting use of your content for AI training and grounding in some other Google systems. It's about use, not about whether your pages appear in Search.

What should I spend time on instead?

The fundamentals that every operator's documentation shares: crawlable pages, content in the HTML, working links, an accurate sitemap and honest structured data.

What to do this week

  • Read your robots.txt for crawler rules you added and forgot.
  • Check that your important pages return real text in the first response.
  • Decide separately whether you want to allow training crawlers.
  • If you publish an llms.txt, treat it as optional and keep it accurate.

Check your own site

Paste your address below. The free scan looks at robots.txt, your sitemap, access for the main AI crawlers and structured data, and shows what it found in seconds. It tells you what a machine can fetch and read. It doesn't predict rankings or citations.

Free scan · No account needed · Results in seconds

Sources

Keep reading

Want to check one file yourself? Try the free AI crawler robots.txt checker or the llms.txt validator, or read how the score is built.