Instructor: Llms.txt is a standard file,
which is designed to help LLMs and AI agents
to better understand and process website content.
So this file typically is placed
in the root directory of a given website
and provides a concise summary
of the site's most important content and structure
in a machine-readable Markdown format.
So the purpose of this is to give
those AI systems, like ChatGPT, LLMs, like Google Gemini,
or your application-based AI agents
to process the web content
more accurately and more efficiently.
So the format is usually a Markdown file,
and the content will include the URLs
to the important pages of the website
with a brief description of each page content
and the purpose of that page.
And it may have an optional additional information.
Now, the benefits of using llms.txt,
it will improve the AI accuracy
when extracting the information from the website,
so to downstream that information makes it very, very easy.
It also enhances the content discoverability
of LLMs with the website content,
so the website owners have an incentive
to really create this file.
It also provides obviously better context
for understanding the website structure
and can even improve the SEO
by making the content more accessible
for AI-driven search engines.
And while it's not an official standard,
llms.txt is gaining massive popularity
in the GenAI community.
The page we are seeing right now
is in the LangGraph official documentation.
You can simply head up there and go to llms.txt,
and the LangChain team has even taken this one step ahead.
So they have two types of llms.txts.
One is a regular one, like we saw earlier in the example,
and the other is llms-full.txt.
Now, they have both for the LangGraph documentation
in Python and also in JavaScript.
So what's the difference?
The difference is that the llms.txt
is only going to include the URLs and a short description
while the llms-full.txt is going to be all the information
and all the texts of those pages.
So, it's going to be a huge file.
Now let's talk about the use cases
and when to use the llms.txt
and when to use the llms-full.txt.
So I'll use the llms.txt, the brief one,
when, for example, I have an AI agent or an MCP server,
which has the web scraping tool, like Firecrawl,
which enables us to download the content of a website.
So once we have it,
we have in the context all the mapping of the website,
and we know exactly which one to download.
So in case, for example,
we need to download the content about LangChain memory,
then we can simply go and give it in the context,
and the LLM would be able to choose the tool
with the correct URL to download only the memory.
So we'll download,
and we'll retrieve only the information that we'll need,
kind of similar like Rack.
And this is very cool,
and this really enhances the quality of agents
which can get information in real time,
if those websites have llms.txt implementation in them.
And don't worry if it sounds too abstract.
In the next video,
I'm going to show you hands-on
how to go and how to use llms.txt and how to use MCP
in order to get really good information
and really enhance your application's answers.
And it's going to fetch that information in real time,
so this is the key point here.
Now, the llms-full.txt,
it's going to have much more information.
It's going to be a huge file with all the information in it.
Now, when and how to use that,
we can simply download it,
and if we'll chunk it ourself
and index it into our vector store,
we can get very similar functionality.
If we're using LLMs with large context,
we can even send the entire thing.
Or if those LLMs have context cache,
we can cache this information.
Now, there are advantages and disadvantages
for using llms.txt and using llms-full.txt.
So if we want to use the abbreviated version, the llms.txt,
with only the brief information and the links,
then the usage pattern of this is usually
by using an agent with a scraping tool and a search tool.
So once we do that,
we are going to get real time information,
so that's one advantage.
However, it's going to take us a bit more time
because we first need to download this page
and to fetch it with the URL scraper.
We then need to process it with an LLM,
then the LLM will tell us what we need to download.
So then we need to make another fetch
and to download another file
and another web page content and to process that,
and only then we'll get a result.
So while we're getting realtime information,
because it's going to fetch the information dynamically,
the latency is going to be a bit higher.
And by the way, don't freak out
about the demo I'm showing you right now,
we're going to implement this hands-on in the next video.
So it's really cool,
and it's a very cool pattern you should be familiar with,
and it's going to use MCP.