llms.txt is a plain markdown file you publish at yoursite.com/llms.txt to give AI systems a curated map of your most important pages, instead of making them parse cluttered HTML. Anthropic’s Claude and a cluster of developer tools read it. Google has said, repeatedly, that it does not and has no plans to. Adoption is climbing, but most published files sit unread.
Key takeaways
- llms.txt is a markdown file at the domain root that lists a site’s key pages for AI agents to read — it does not control crawling the way
robots.txtdoes. - Jeremy Howard of Answer.AI proposed the format on September 3, 2024; the spec lives at llmstxt.org, 2024.
- An Ahrefs scan of 137,210 domains found 28% now publish a valid llms.txt file, but 97% of those files get zero AI requests (Ahrefs, 2026).
- Google’s Gary Illyes compared it to the 1990s meta keywords tag; a separate 300,000-domain study found no measurable link between having the file and getting cited in AI answers (Search Engine Journal, 2025).
- Adding one costs an afternoon and breaks nothing — the honest case for it is housekeeping, not a ranking lever.
What exactly is in an llms.txt file?
An llms.txt file is a markdown document, not HTML: an H1 with your site or project name, a one-line blockquote summary, and H2 sections that list links to your most important pages with a short description under each. The idea, per the original specification, 2024, is that an AI agent working with limited context can read this one page instead of crawling and parsing your whole navigation, ads and scripts to figure out what your site is about.
A companion file, llms-full.txt, goes further and inlines the full markdown text of those pages in one place, built by Mintlify with Anthropic and later folded into the same proposal. Both files are advisory. Nothing forces an AI system to open them, and nothing stops a site from writing one that describes pages it doesn’t actually have.
Where it fits next to the files you already have
Every reasonably maintained site already has robots.txt (crawl permissions) and sitemap.xml (a complete, machine-readable list of URLs for search engines). llms.txt is neither. It’s a curated, human-edited shortlist aimed at a different reader — an AI model, not a search index — and it carries no enforcement mechanism at all.
How is llms.txt different from robots.txt and sitemap.xml?
The three files answer three different questions, and mixing them up is the most common mistake we see when a site adds llms.txt expecting it to behave like the other two.
| File | Question it answers | Who reads it | Enforceable? |
|---|---|---|---|
| robots.txt | What are you allowed to crawl here? | Search and AI crawlers that choose to respect it | Advisory, widely honored by major crawlers |
| sitemap.xml | What are every one of my URLs? | Search engine indexers | Advisory, used for discovery, not ranking |
| llms.txt | What should an AI agent read first to understand this site? | A short list of AI tools and agent frameworks; not Google | No enforcement — purely a suggestion, unverified |
Do AI models actually read llms.txt files?
Some do; most don’t, and the ones that do rarely use it for the thing site owners hope for. Ahrefs’ scan of 137,210 domains in May 2026 found that of the requests that do hit a published llms.txt file, only 19.5% come from AI bots at all — SEO audit tools account for a larger share (21.7%) than actual AI retrieval traffic. PerplexityBot and similar AI search crawlers made up just 1.1% of total requests (Ahrefs, 2026).
On the explicit “does Google use this” question, the answer is no, and Google has said so more than once. Gary Illyes wrote on Bluesky that the comparison to the 1990s meta keywords tag is “very easy to draw,” adding that everyone remembers how useful that tag turned out to be (Search Engine Roundtable, 2025). John Mueller went further the following year, saying the format “was never built for that and won’t deliver it,” after one of the standard’s own creators told him it was designed to help an AI that already knows a site find more of its pages — not to get a site discovered in the first place (Shopifreaks, 2026).
Does llms.txt help you get cited in AI answers?
The closest thing to a direct test says no, at least not yet. A study spanning roughly 300,000 domains found that only 10.13% had an llms.txt file at all, and when researchers fed “has llms.txt” into a model predicting AI citation frequency, removing that feature actually made the model more accurate — the file wasn’t just unhelpful, it was noise (Search Engine Journal, 2025). That lines up with what we’d expect from the usage numbers above: a file that 97% of AI crawlers never request can’t be doing much to shape what those same systems cite.
What still moves AI citations, based on that same body of research and on our own guide to AI Overviews work, is the ordinary stuff: clear, direct answers near the top of a page, structured data, and a page that already ranks well on its own HTML. llms.txt doesn’t substitute for any of that.
Should your site have an llms.txt file?
Having one is low-risk and low-reward, which is a reasonable thing to decide on once rather than relitigate every quarter. Here’s the order we’d actually do it in:
- Finish the things with evidence behind them first — schema markup, a direct answer in your first paragraph, and clean crawlability. None of that is optional the way llms.txt is.
- Write the file by hand, not by generator. List only pages that exist, with a one-line description that matches the page, in the H1/blockquote/H2-list format from the specification, 2024.
- Keep it short. A handful of your highest-value pages beats a dump of your whole sitemap — the entire point is a curated shortlist, not a second sitemap.xml.
- Publish it at the domain root (
/llms.txt), not buried in a subfolder, so the handful of tools that do check for it find it on the first look. - Don’t touch anything else for it. It needs no change to robots.txt, no new redirect, and no budget beyond the hour it takes to write.
- Re-check it twice a year, same cadence as your sitemap, so it doesn’t quietly describe pages you’ve since removed.
If you’re weighing this against other fixes and you’re not sure which one actually moves the needle for your site, that triage question — what’s worth doing now versus what’s a five-minute afterthought — is exactly what we sort out in TheMarkElite’s free 30-minute diagnostic, with a prioritized list back to you within 48 hours (book one here).
Frequently asked questions
What does llms.txt actually do?
It gives AI tools that choose to check for it a short, curated list of your most important pages in markdown, so they can understand your site without parsing full HTML. It doesn’t block or allow crawling, and it doesn’t guarantee any AI system will read it.
Is llms.txt the same as robots.txt?
No. robots.txt sets crawl permissions and is widely respected by major crawlers. llms.txt is a curated content summary with no enforcement at all — a site can publish one and still have it ignored entirely.
Does Google use llms.txt for AI Overviews?
No. Google representatives have said this directly, comparing llms.txt to the old meta keywords tag and confirming AI Overviews rely on standard SEO signals, not llms.txt (Search Engine Roundtable, 2025).
Will adding llms.txt improve my AI search citations?
Current research says there’s no measurable lift. A 300,000-domain study found no link between having the file and citation frequency (Search Engine Journal, 2025), and Ahrefs found 97% of published files get zero AI requests to begin with (Ahrefs, 2026).
How do I create an llms.txt file?
Write a markdown file by hand with an H1 for your site name, a one-line blockquote summary, and H2 sections linking to your highest-value pages with short descriptions, following the format at llmstxt.org, 2024. Publish it at your domain root as /llms.txt.
Is llms.txt required for SEO?
No. It isn’t part of how Google ranks or surfaces pages, and the evidence so far doesn’t show it affecting AI citations either. It’s optional housekeeping, worth doing after the fundamentals, not instead of them.
Most of what determines whether an AI system ever mentions your site has nothing to do with this one file — it’s whether the page itself answers the question clearly, cites real numbers, and is built so Google and an AI model can both parse it. If you want a straight read on where your own site stands on that, TheMarkElite’s free 30-minute diagnostic covers it and comes back to you within 48 hours — request yours here. For more on what actually does move AI visibility, see our insights hub.
Last updated: November 9, 2026. Written by the TheMarkElite team.