llms.txt in 2026: who has one, and who reads it

llms.txt is a text file at the root of a website, /llms.txt, that lists what the site is and which pages matter, in Markdown, for AI assistants and agents. It was proposed in September 2024. Two years later, the questions are the same: who publishes one, and does any AI actually read it?

We measured both. On November 3, 2026, we requested /llms.txt on 3,202 websites: the 1,000 most visited websites, 2,000 websites launched on Hacker News since July, and the 202 launch directories of our catalog. We also read 30 days of Cloudflare logs of two websites: one with an llms.txt, one without. This is what we found:

• New websites have one more often than big ones. 36% of the new websites from Show HN serve an llms.txt file, against 16% of the top 1,000 websites that answered. Launch directories: 55%.
• 10% of new websites only seem to have one. They answer /llms.txt with their HTML home page, so a check that only looks for a "200 OK" counts a file that isn't there.
• The crawlers behind AI search don't ask for it. On findly.tools, which has an llms.txt, GPTBot made 4,000 requests in 30 days, ClaudeBot 15,050 and Googlebot 24,870. None of them requested llms.txt.
• robots.txt is read eleven times more. The same site got 31,990 requests for robots.txt and 2,920 for llms.txt and llms-full.txt together.
• Most llms.txt readers are not AI companies. 7% of the 2,920 requests came from a named AI crawler or agent. The rest: browsers or tools without a bot name, SEO tools, llms.txt registries, scripts and search engines.
• The agents that do read it work for a person. ChatGPT-User and Claude-User fetched the file: they act when someone asks the assistant to look at a site, or when a coding agent reads documentation.

What llms.txt is#

The proposal (opens in a new tab), by Jeremy Howard, describes the file as a way "to provide information to help agents use a website". It is a Markdown file:

• A title, the name of the site, on the first line.
• A short summary in a quote.
• Sections with lists of links to the pages that matter, often to a Markdown version of each page.
• An "Optional" section for the links an agent can skip.

A second file, llms-full.txt, is a common addition: the content of those pages in one file. The proposal is not a standard of any search engine or AI company. Its second version, from August 2026, notes that documentation platforms generate the file automatically, and that the AI labs publish one for their own developer documentation.

How we checked#

• The top 1,000 websites. The first 1,000 domains of the Tranco list, a research ranking of the most popular websites. Many are not websites people visit (content delivery networks, APIs): 726 of them answered.
• New websites. Hacker News lists 7,015 websites on their own domain posted as Show HN between July 1 and October 24, 2026. We took 2,000 at random; 1,954 answered.
• Launch directories. The 202 directories and launch platforms of our catalog, all answered.
• The request. A GET of /llms.txt and /llms-full.txt, following redirects, with a bot name. A file counts as an llms.txt when the server answers 200 at that address with text, not HTML, whose first line is a Markdown title, as the proposal asks.
• The logs. Cloudflare analytics of findly.tools, a tool directory run by the team behind Submitium that publishes an llms.txt and an llms-full.txt, and of submitium.com, which has neither. From October 4 to November 3, 2026.

Cloudflare's analytics are samples: counts are estimates rounded to the nearest 10, and a bot with a few requests can show 0. A user agent says which bot it claims to be; we did not check the IP addresses behind each name.

Who has an llms.txt#

• Large websites. 118 of the top 1,000 publish one, among them Cloudflare, GitHub, Adobe, Etsy, Dropbox and Sentry. The top 100 do less: 8 of the 72 that answered, 11%.
• New websites. More than a third have one, at the same rate every month from July to October 2026: 36%, 33%, 35% and 38%. The file did not go with more attention on Hacker News: 37% of the websites whose post got fewer than 5 points had one, against 30% of those with 20 points or more.
• Launch directories. More than half have one. Directories depend on being found, by people and by AI assistants.

The median file is 13 KB on the large websites and 6 KB on the new ones, with 9 to 16 links in its first 4 KB.

The files that aren't there#

10% of the new websites answer /llms.txt with "200 OK" and an HTML page: the home page of a single-page app, which serves the same page at every address. An SEO tool that checks only the status code counts these as an llms.txt. An AI agent that requests them gets a page of HTML and scripts. To check yours, open /llms.txt on your domain in a browser: you should see plain text, not your website.

Who reads it#

On findly.tools, the crawlers of AI and search companies made hundreds of thousands of requests in 30 days. Their requests for llms.txt:

• None from GPTBot (OpenAI's crawler for training), ClaudeBot (Anthropic's), Googlebot, bingbot, Applebot and Amazonbot.
• 10 each from OAI-SearchBot (ChatGPT's search crawler, 79,600 requests in all) and PerplexityBot (21,550 in all).
• 90 from meta-webindexer, Meta's crawler, out of 73,550 requests.
• 40 from ChatGPT-User and 10 from Claude-User, agents that fetch a page when a person asks them to, and 20 from GrokBot.

Google already said it plainly in its guide to AI features (opens in a new tab): "You don't need to create new machine readable files, AI text files, or markup to appear in these features." Its crawler behaved accordingly.

Who did request it? Mostly tools that don't say who they are: 57% of the requests came with an ordinary browser user agent, and 14% from scripts such as curl or Python. 16% came from named bots that are not AI companies: SEO tools, directories, and registries that list llms.txt files. Some of the browser requests may be people checking the file, or AI tools that don't announce themselves.

On submitium.com, which has no llms.txt, /llms.txt was requested about 30 times in 30 days: by a crawler named Mydentify and two browser user agents. No AI company's bot asked for it. The same bots read its robots.txt: ClaudeBot about 310 times, Googlebot 180, OAI-SearchBot 30.

Should you add an llms.txt?#

It costs an hour, and it does no harm. Chrome's Lighthouse has an llms.txt audit (opens in a new tab): without the file, it marks the audit "Not Applicable (N/A), as providing the file is optional at the moment". Nothing in our data shows it brings visits or citations.

• Add one if your product has documentation, an API or a command-line tool. Coding agents and assistants read documentation for their users, and a clean summary with links to Markdown pages saves them work. Claude-User, from Claude Code, fetched findly.tools' llms-full.txt during our 30 days.
• Don't count on it to appear in ChatGPT, Perplexity or Google. The crawlers that feed their search read your pages and your robots.txt.
• Do these first:
  • Let the AI search crawlers in: OAI-SearchBot, PerplexityBot, and the others you want, in robots.txt.
  • Make your pages readable without JavaScript, so a crawler that doesn't run scripts sees the text.
  • Publish the pages AI assistants cite. In our study of what 200 AI answers cite, comparison pages and product pages were 68% of the sources.

If you write one#

# Your product

> What the product does, for whom, in one sentence.

## Docs

- [Quick start](https://<your domain>/docs/start.md)
- [API reference](https://<your domain>/docs/api.md)

## Pricing

- [Plans and limits](https://<your domain>/pricing.md)

## Optional

- [Changelog](https://<your domain>/changelog.md)

• Serve it as text, at /llms.txt, with a status 200. Check that the address doesn't return your HTML app.
• Say what the product does, what it costs, and what it doesn't do. findly.tools' file lists what the site does not offer, so that agents don't invent it.
• Link to pages that exist, ideally in Markdown, and update the file when they change.
• Keep it short. The median file in our sample is 6 to 13 KB.

How to see who reads it#

In Cloudflare, Analytics, filter the requests by path /llms.txt and look at the user agents. On your own server, search the access log for "llms.txt". Compare with robots.txt over the same period: the ratio tells you more than the total.

Where Submitium fits#

More than half of the launch directories in our catalog publish an llms.txt: they want to be found by people and by AI assistants. A listing on them gives your product a page that both can read, and a link to your site.

We submit your product by hand to directories and launch platforms that give a free dofollow link, keep their badges in one badge wall on your site, and check the link on each live listing six times over 90 days. A directory costs $0.99, less from the 51st, and a first order starts at 30 directories. See the pricing and the full list of directories.

Questions#

What is llms.txt?#

A Markdown file at the root of a website, /llms.txt, with the name of the site, a summary and links to its main pages, written for AI assistants and agents. It was proposed by Jeremy Howard in 2024. It is a proposal, not a standard of Google or of an AI company.

Does llms.txt work?#

It is read, but rarely by the crawlers behind AI search. On a website that has one, GPTBot, ClaudeBot and Googlebot didn't request it in 30 days, while OAI-SearchBot and PerplexityBot requested it about 10 times each. Agents acting for a user, such as ChatGPT-User and Claude-User, did read it.

Does Google use llms.txt?#

Google says no special file is needed for its AI features: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." Googlebot made 24,870 requests to findly.tools in 30 days and none to its llms.txt.

Does ChatGPT read llms.txt?#

ChatGPT-User, which fetches pages when a user asks ChatGPT to, requested findly.tools' llms.txt and llms-full.txt 40 times in 30 days. OAI-SearchBot, the crawler for ChatGPT's search, requested it about 10 times out of 79,600 requests. GPTBot, the training crawler, never did.

Should I add llms.txt to my website?#

If you have documentation, an API or a developer tool, yes: it helps agents that read docs for their users. For a marketing website, it does no harm, but put robots.txt, pages readable without JavaScript, and comparison pages first.

What is the difference between llms.txt and robots.txt?#

robots.txt tells crawlers what they may visit, and every major crawler reads it. llms.txt describes the site for AI agents and blocks nothing. On findly.tools, robots.txt was requested 31,990 times in 30 days and llms.txt 1,910 times.

What is llms-full.txt?#

A companion file with the full text of the pages listed in llms.txt, in one Markdown file. 11% of the new websites and 23% of the launch directories we checked publish one.

How do I check that my llms.txt works?#

Open /llms.txt on your domain: you should see plain text that starts with # and your site's name, not your home page. Then check the links in it, and your logs to see who requests it.