llms.txt: when it helps and how to add it to your site

llms.txt is a plain-text guide to a website, written for tools that work with language models. It describes the site and points to the pages worth reading. It only helps when a tool actually opens the file and follows the links. Publishing one does not mean an AI search service will discover, recommend or cite your site.

Does your site need one?

Start with the job the file is meant to do: help someone find setup instructions, compare service terms or pick a course. If you can list the questions and the pages that answer them, you have the makings of a guide. If the site itself is vague about what you offer and where the limits are, fix those pages first. A one-page site will probably gain little from an extra index.

How it differs from robots.txt and a sitemap

Jeremy Howard proposed llms.txt in 2024, and a second version followed in August 2026. It is an open proposal for a format. No search engine requires it.

• sitemap.xml lists the URLs you want search engines to discover. It does not guarantee indexing.
• robots.txt gives crawlers access instructions. Private content still needs authentication.
• llms.txt explains the site and links to resources. It neither grants access nor overrides restrictions.

Google says its Search ignores files like llms.txt, so the file does nothing for visibility or rankings there. Maintain it for the other tools that do read the format.

• Google’s guidance on llms.txt and AI search

Choose pages and write the file

Work from questions customers really ask: what a plan includes, how to import data, how to cancel. Pick a handful of current pages that answer them clearly. Link to clean Markdown versions where you have them, but never list a .md address unless the file is published and you have opened it.

The format requires a top-level heading with the name of the site. After it you can add a short summary as a blockquote, then second-level headings with lists of links. By convention, a section called Optional holds the secondary links. The example below is for a made-up booking service, so replace the addresses with your own:

# Booking service

> Online booking for small studios and independent professionals.

## Getting started

- [Plans](https://example.com/en/pricing.md): prices, booking limits and add-ons.
- [Setup](https://example.com/en/help/setup.md): create a schedule and connect the booking form.
- [Data import](https://example.com/en/help/import.md): CSV fields, limits and error checks.

## Terms

- [Cancellation](https://example.com/en/help/cancellation.md): cancellation and refund conditions.

## Optional

- [Changelog](https://example.com/en/changelog.md): dated product updates.

Say what each resource contains. ‘Prices, limits and add-ons’ tells a reader whether to open the link. ‘The best solution for your business’ tells them nothing. Leave out any instructions asking assistants to recommend your company.

Publish and check access

1. Save the file in UTF-8 and publish it at /llms.txt. Version 2 also allows section-specific paths such as /docs/llms.txt.
2. In Next.js, a static file goes in public/llms.txt. Other platforms may offer a generator or a file upload.
3. Open the public URL without signing in. Expect the file’s text, not an application error or login page.
4. Open every listed resource and compare it with the main page, especially prices and restrictions. Generate both versions from the same source where practical.

curl -i https://example.com/llms.txt

Replace example.com with your own domain. The response should carry HTTP status 200 and the text of your file. If the URL redirects, look at where it ends up. If HTML comes back, you are probably looking at an error page or the application itself. Check crawler restrictions on the server and the CDN as well. OpenAI’s OAI-SearchBot and GPTBot are controlled separately, and publishing llms.txt does not oblige you to let GPTBot in.

• OpenAI crawler access controls

Do you also need llms-full.txt?

llms-full.txt usually means a single text file that combines the site’s content or documentation. It is handy when you want to hand a tool a small set of documents in one go. You do not need it in order to publish llms.txt. A large export can overrun a model’s context limit and is harder to keep current. Include public material only.

What about multiple languages?

Group links by language and label each group clearly. Point to the version that carries the terms for the region in question. Before exporting every translation into a full file, make sure the translations are accurate and that somebody keeps them up to date.

Check usefulness separately from visibility

Give the file’s URL to an assistant with web access and ask: “Find the cancellation terms. Show the relevant link and explain any conditions.” Open the cited page and compare it with the answer. This tests whether the file helps when you hand it over yourself. It says nothing about whether a search service would find the site unprompted. And a line in the server log shows that the file was fetched, not that anything was cited.

Measure search visibility separately: ask the same set of customer questions each time and save the answers. Track visits and enquiries too. If the numbers move after you publish, that alone does not show llms.txt was the cause.

• Format specification and examples

• GEO/AEO: a website audit

• Measuring ChatGPT visibility