Every few months, a new idea promises to help websites stand out in AI search. One of the most talked-about is a small text file called llms.txt.
It sounds simple: add a file to your site that tells AI tools which pages matter most. SEO plugins now generate it in a few clicks, and many marketers are asking whether they are falling behind without one.
The honest answer is more nuanced. An llms.txt file is cheap to create and unlikely to cause harm, but there is limited public evidence that major AI search engines use it today.
That makes it worth understanding clearly before you spend time on it. It also helps to see where it fits within a broader generative engine optimization strategy.
In this guide, you’ll learn what llms.txt is, how the format works, what search engines have said about it, how to set it up in Rank Math, and whether your website actually needs one.
What Is llms.txt?
llms.txt is a proposed standard for a Markdown file placed at the root of a website (for example, yoursite.com/llms.txt). It gives large language models a short summary of the site and a curated list of links to its most useful pages, so AI tools can find key content without parsing every page.
The idea was proposed by Jeremy Howard in September 2024 and is documented at llmstxt.org.
It is a proposal, not an official web standard. No search engine or AI company is required to read it.
Why Was llms.txt Created?
Web pages are built for people and browsers. AI tools that read them on demand face a few practical problems.
Limited Context Windows
Language models can only process a certain amount of text at once. A full website is usually far too large to read in one go, so a tool needs to know where to start.
Cluttered HTML
Navigation menus, cookie banners, scripts and ads all sit alongside the actual content. Converting that into clean text is error-prone.
Use at Inference Time
The proposal focuses mainly on inference: the moment an AI assistant fetches information to help a user. It is less about training data, although the author notes that training could also make use of it.
Imagine a hypothetical developer asking an AI coding assistant how to use a payment API. If the API’s documentation site has an llms.txt file, the assistant could read one short index, jump straight to the authentication and quick-start pages, and skip the marketing pages entirely.
That scenario is where llms.txt makes the most sense: tools that are explicitly told to read a specific site.
How Is an llms.txt File Structured?
The format uses plain Markdown in a specific order. Only the first element is required.
A Required H1 Title
The file starts with a single H1 heading containing the name of the project or website.
A Short Summary
Next comes a blockquote with a brief description of what the site covers. This is the part an AI tool reads first, so it should be precise.
Optional Detail Sections
You can add paragraphs or lists with extra context, such as who the site is for or how its content is organized. These sections should not use headings.
File List Sections
H2 sections then group links. Each link is written as a Markdown link, optionally followed by a colon and a short note.
The “Optional” Section
By convention, an H2 called “Optional” holds secondary links that a tool can skip if it needs a shorter context.
A simplified example for a hypothetical marketing blog might look like this:
# Acme Marketing Blog
> Practical guides on SEO and AI search for small business owners.
## Guides
– [What Is GEO?](https://example.com/what-is-geo/): Beginner introduction
– [Schema Basics](https://example.com/schema/): Structured data walkthrough
## Optional
– [About Us](https://example.com/about/)
The file is meant to be readable by both people and machines. If you can scan it in a minute, it is probably the right length.
What Are llms-full.txt and Markdown Page Versions?
Two related ideas often appear alongside llms.txt.
- Markdown page versions: the proposal suggests offering a clean Markdown copy of a page at the same URL with “.md” added. This gives AI tools a clutter-free version of each document.
- llms-full.txt: some documentation platforms publish a second file containing the full text of all key pages in one place. This is a common convention, but it is not part of the core proposal.
For most content websites and blogs, a standard llms.txt file is enough. The full-text and Markdown variants are mainly used by software documentation sites.
llms.txt vs robots.txt vs sitemap.xml
These three files are easy to confuse because they all sit at the root of a site. They do different jobs.
| File | Main Purpose | Audience | Status |
|---|---|---|---|
| robots.txt | Tells crawlers which URLs they may access | Search and AI crawlers | Widely supported standard |
| sitemap.xml | Lists URLs you want discovered and indexed | Search engines | Widely supported standard |
| llms.txt | Summarizes the site and points to key pages | AI tools and agents | Proposal with limited confirmed support |
The most important difference: llms.txt does not control access. If you want to allow or block AI crawlers, you still need robots.txt. Google explains how that file works in its introduction to robots.txt.
Do AI Search Engines Actually Use llms.txt?
This is the question that matters most, and the evidence so far is thin.
What Google Has Said
Google’s search representatives have said publicly that Google Search does not use llms.txt. John Mueller has compared it to the old keywords meta tag, which search engines stopped trusting because site owners could write anything in it.
Google has also said that normal SEO practices are what matter for appearing in AI Overviews.
What AI Companies Have Said
At the time of writing, the major AI search providers have not published clear documentation confirming that they use llms.txt to choose sources or citations.
Some AI companies publish llms.txt files on their own documentation sites. That shows they find the format useful for developers, but it is not the same as a commitment to read yours.
What Server Logs Show
Reports from site owners are mixed. Some say they have seen AI crawlers request the file, while others report very few requests at all.
The safest conclusion is that llms.txt is not a ranking or citation signal you can rely on today. It may become more useful if adoption grows.
Does Your Website Need an llms.txt File?
For most websites, llms.txt is a “nice to have”, not a priority. Whether it is worth doing depends on your site and how much time it takes.
When It Can Make Sense
- Your SEO plugin can generate it in a few minutes
- You run a documentation, developer or software help site
- Your audience uses AI coding or research assistants that read specific sites
- You want a clean, curated index of your best content anyway
When It Should Wait
- Your core pages have thin content or unclear structure
- Your robots.txt, sitemap or indexing has unresolved problems
- You have no structured data or clear author information yet
- Creating the file would take developer time you need elsewhere
Consider a hypothetical local bakery with ten pages. Adding llms.txt through its SEO plugin takes five minutes and costs nothing. But updating its Google Business Profile, opening hours and menu page would almost certainly do more for its AI visibility.
How to Create llms.txt in Rank Math
Recent versions of Rank Math include a built-in llms.txt module. Rank Math documents the feature in its llms.txt guide.
- Update Rank Math to the latest version.
- Go to Rank Math SEO → Dashboard and switch on the LLMS Txt module.
- Open the module settings (or go to General Settings → Edit llms.txt).
- Choose which post types to include, such as Posts and Pages.
- Optionally include taxonomies such as Categories.
- Save your changes, then visit yoursite.com/llms.txt to check the output.
Rank Math excludes noindexed content automatically, which helps keep low-value pages out of the file.
If your site uses a caching plugin, purge the cache after enabling the module. Otherwise, visitors and crawlers may still see an old “page not found” response.
How to Create an llms.txt File Manually
If you do not use a plugin with this feature, you can write the file yourself.
- Create a plain text file named llms.txt.
- Add an H1 line with your site name.
- Add a one or two sentence summary in a blockquote.
- Group your most important pages under H2 headings.
- Write each link as a Markdown link with a short description.
- Upload the file to your website’s root folder.
- Check that it loads at yoursite.com/llms.txt as plain text.
Keep it short and curated. The point is to highlight your best pages, not to copy your sitemap.
What Should You Include in llms.txt?
Treat the file like a reading list for a new colleague. Which pages would you hand them first?
- Pillar guides that explain your main topics
- Product, service or documentation overview pages
- Pricing and contact pages, if they are relevant to users’ questions
- About and author pages that explain who is behind the site
- Glossaries or FAQ pages that define key terms
What to Leave Out
- Thin tag and archive pages
- Outdated posts you no longer maintain
- Thank-you, login and checkout pages
- Duplicate or near-duplicate content
A short description next to each link helps. Write it the way you would explain the page to a person, without keyword stuffing.
How to Check That Your llms.txt Is Working
- Open yoursite.com/llms.txt in a private browser window.
- Confirm it returns the file, not a 404 error or your theme’s “page not found” page.
- Check that the links work and point to live, indexable pages.
- Purge your site and CDN cache if you see an old version.
- Review your server logs over the following weeks to see which bots, if any, request it.
Log data is the only reliable way to know whether AI crawlers are reading your file. Treat what you find as a signal of interest, not proof of influence on citations.
Common llms.txt Mistakes to Avoid
- Expecting it to boost rankings or citations on its own
- Using it to block AI crawlers: llms.txt has no access-control role
- Listing every URL instead of a curated selection
- Including noindexed or broken pages
- Stuffing keywords into the summary and link descriptions
- Forgetting to update it when you publish or remove key pages
- Skipping the basics: crawlability, structure and authority still matter far more
Where llms.txt Fits in Your GEO Strategy
llms.txt is a small, optional layer. The factors that tend to influence AI visibility are much broader.
AI search tools typically retrieve content through search indexes and then choose passages that answer a question well. The guide on how retrieval-augmented generation chooses sources explains this process in more detail.
That means your priorities usually look like this:
- Make sure search engines and AI crawlers can access and index your pages
- Structure content so answers are easy to extract
- Build authority through expertise and third-party mentions
- Add structured data that clarifies your content and brand
- Then add optional extras such as llms.txt
For practical formatting advice, see the guide on how to structure content for AI search. If you are working with a tight budget, these free GEO tools for small businesses can help you cover the essentials first.
llms.txt Checklist
- Core SEO basics (indexing, sitemap, robots.txt) are in good shape
- File created through your SEO plugin or written manually
- Starts with an H1 site name and a short summary
- Lists only your most useful, indexable pages
- Each link has a short, plain-language description
- Loads at /llms.txt without a 404 after cache purging
- Review date set for updating the file
- Server logs checked for AI crawler requests
Final Takeaway
llms.txt is a sensible idea: a short, human-readable map that helps AI tools find your most useful content.
But it is still a proposal. Google has said its search systems do not use it, and major AI search providers have not confirmed that it affects which sources they cite.
If your SEO plugin can generate the file in a few minutes, there is little downside to adding it. Just keep your expectations realistic and put most of your effort into content quality, structure and authority.
If you’re unsure where llms.txt and other technical fixes rank on your list, our team can help with a GEO audit.
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed Markdown file placed at the root of a website. It gives AI tools a short summary of the site and a curated list of links to its most useful pages, so they can find key content without reading every page.
Does Google use llms.txt?
Google’s search representatives have said that Google Search does not use llms.txt. Google advises that normal SEO practices are what matter for appearing in AI Overviews.
Does llms.txt help you get cited by ChatGPT or Perplexity?
There is no public confirmation that major AI search providers use llms.txt to choose sources or citations. It may help tools that are directed to read a specific site, but it should not be treated as a citation signal.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which URLs they may access, while llms.txt summarizes your site and points to key pages. llms.txt cannot allow or block crawlers.
How do I create llms.txt in Rank Math?
Update Rank Math, enable the LLMS Txt module from the Rank Math dashboard, then choose which post types and taxonomies to include in its settings. Save the settings and check the file at yoursite.com/llms.txt.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short index of links with descriptions. llms-full.txt is a separate convention used by some documentation sites that contains the full text of key pages in a single file.
Can llms.txt hurt my SEO?
There is no evidence that a well-formed llms.txt file harms search rankings. The main risk is spending time on it instead of higher-impact work, or listing outdated and low-quality pages.
Should a small business add llms.txt?
If your SEO plugin can create it in a few minutes, it is a reasonable low-effort addition. Most small businesses will see more benefit from improving their content, local listings and structured data first.