Does Our Website Need an llms.txt File To Get Cited By AI?

With more and more people using AI tools like ChatGPT, and Google's extended use of AI Overviews presented to users of its search engine, you might be wondering if your business or charity needs to implement an llms.txt file to give it a discovery boost.

‍

Why is AI Discoverability Important?

‍

Firstly, let's explore why AI discoverability should be on your radar at all.

‍

The way we search for things online is changing rapidly thanks to the access we now have to AI tools like ChatGPT, Google Gemini, Microsoft Co-Pilot, and AI Overviews in Google's search engine.

‍

In the past, we'd type a few words or maybe a short sentence into a search engine, and then skim through the links to websites ordered in the results to find the ones that seemed to best answer our search queries or search intent. Then we'd click on some of those links to see if the website on the other end satisfied our searches. 

‍

Now, many of us are instead going to AI tools and posing our queries there, and rather than just entering a few keywords or short sentences, we're using lengthy and detailed prompts. These longer prompts are also being used in Google searches. 

‍

In doing so, we're essentially giving these tools much more precise inputs to help fine tune and narrow our searches, and then letting them (a) find what we're looking for, (b) sift through the results for us, and (c) supply us with a summary of those results.

‍

And that's why AI discoverability matters. Because, already, if ChatGPT gives someone three links to its sources and your website isn't among them, you're invisible. Similarly, if people are happy with the summary AI gives them, they may not click out to your website even when a link is featured in AI Overviews.

‍

In the search landscape of the future, being cited and recommended by AI is going to become increasingly important.

‍

What is an LLMS.TXT File?

‍

An llms.txt file is a plain text or markdown file and is designed specifically to guide AI tools towards a website's most important content. 

‍

Think of it as a way of helping Large Language Models (LLMs) to understand the content of your website so that it's better able to match it to people's prompts.

‍

It's similar to an xml sitemap which most websites use. But where a conventional sitemap will list all pages for search engine crawlers to index, an llms.txt file focuses on core content.

‍

For instance, if you sell products, your llms.txt file would be used to draw attention to the benefits and features of your products, because it's these you want people to know about when using an AI tool for product recommendations, but it might not include your 'About Us' page because it's less directly relevant.

‍

An llms.txt file is designed to provide AI tools with context and enable them to extract information more quickly. 

‍

Will An LLMS.TXT File Get My Website Links Into AI Responses? 

‍

To answer this, we thought we'd ask the experts.

‍

So, we used ChatGPT and Google Gemini to see what they each had to say on the subject 😆

‍

Here's what they told us.

‍

‍ChatGPT on llms.txt files: Yes, having an llms.txt file can improve how AI tools (like ChatGPT, Perplexity, Claude, etc.) discover and interact with your website, but it’s still an emerging practice. In short: it won’t guarantee better visibility yet, but it sends the right signals and positions your site well as AI discovery evolves.

‍

‍Google Gemini on llms.txt files: While the concept is promising, its real-world impact is still evolving. Some major AI teams are reportedly starting to crawl and use these files, suggesting that for some platforms, it can indeed lead to better visibility and more accurate citations. However, until it becomes a widely adopted standard like robots.txt or sitemap.xml, its benefits won't be universal across all AI tools. For now, it's a forward-thinking strategy that can offer a competitive edge for websites with large, complex, or frequently updated content.

‍

Our current view?

‍

Right now, llms.txt files are experimental, and whilst there's some anecdotal evidence of them helping websites get into AI results, there's no empirical evidence of it making a huge and consistent difference. You should be wary of anyone claiming that an llms.txt file is a guaranteed way to get cited by AI tools.

‍

However, given the direction of travel, it's probably worth adding one to your website. 

‍

There's certainly no obvious reason not to, and it also gives you some control over how you show up in AI responses too.

‍

Apart From Using an LLMS.TXT File, How Else Can We Get Our Website Into AI Results?

‍

Whilst llms.txt files are in their infancy, you can improve your chances of getting your website and content cited by AI tools by doubling down on your existing SEO practices.

• Use Schema Markup on every page. This 'structured data' helps search engine crawlers and LLMs to understand the content of your website.

• Build lots of internal links. Group content in clusters and link them so that it's obvious there's breadth and depth to your website. For instance, link from product pages to relevant blogs, and from your blogs to relevant case studies. It's a way of establishing topical authority.

• Use H1 and H2 title tags. Give every page a H1 title, and then break up your content into sections that are each headlined with a H2 subtitle. This makes it easy for search engines and LLMs to parse your content, but also makes it easier for people to read and absorb what you're saying.

• Make good use of bullet points. FAQ-style bullet lists are particularly suited to being used as short, concise summaries by AI tools.

• Rewrite some of your core content for AI. Thinking about how people are using AI tools (conversational prompts looking for answers in an easy-to-understand format) examine your core content and explore how you could write it specifically for AI. For instance, could you offer a short bullet-point summary of your services written in the form of an answer to a ChatGPT prompt, but where it's still user-friendly for people who visit your website? 

‍

Key Takeaways: Should Your Website Have An LLMS.TXT File For AI Discoverability?

‍

• An llms.txt file is designed to make it easier for AI tools to understand the core content of websites. 

• Currently, the use of llms.txt files is a new and emerging practice, and not all AI tools recognise them yet.

• Having an llms.txt file on your website doesn’t guarantee that your content will be cited by tools like ChatGPT and others. 

• Whilst there is no guarantee that using an llms.txt file will get your website into AI responses, it will make it easier for Large Language Models (LLMs) to understand and extract relevant content from your website, which could at least improve your chances of showing up.

• Right now, the use of an llms.txt file is probably best seen as a way to futureproof your website as the use of AI tools evolves.

• Alongside the use of an llms.txt file, you can improve the chances of your website being cited in AI responses by doubling-down on existing technical and on-page SEO practices.

‍