In the next couple of videos, we'll be chunking up the link chain documentation into smaller chunks,
so we'll be able to provide it as context to our LMS.
We are going to then embed.
So we're going to turn those chunks into vectors.
And that vectors are going to be indexed into our vector store.
All right.
So here we have the snippet of the chunking face.
And we start by logging everything.
And the entire chunking process is going to be with the recursive character text filter from name chain.
We're going to provide it with the chunk size of 4000 chunk over up of 200, and it's going to recursively
chunk up our documents.
Now, I do elaborate on the recursive character text splitter in this video over here if you want to
check it out.
But the overall idea here is that it's going to semantically chunk up our documents.
So first it's going to chunk it up by paragraphs, then by new line.
And we want to limit the chunk sizes to 4000 and the chunk overlap to be 200, and those are 4000 characters
and 200 characters.
So after we have the text splitter object ready, we simply use the built in method of split documents
and we provide it with the link chain documents.
Once it's finished, we're going to get back a larger list of documents.
But the documents are going to be chunked up and then we are going to log everything.
So this entire step in the pipeline is very simple to implement.
Link chain is going to do all the heavy lifting for us.
And I just want to mention a few things.
First of all, this isn't a silver bullet chunking method.
So there are many chunking methods and this is a very deep topic.
If you want to go and explore it, there are many strategies small to big semantic chunking and a lot
of cool things you can do in this phase for optimizations.
This is the first thing I wanted to mention.
And second thing I want to say is that because LMS now have larger token limits which can reach these
days in 2025 to 1 million tokens like in a anthropic, and can go to Gemini 2.5 with 2 million input
tokens.
Then this chunking over here that we were doing is important, but it's not the thing we should be focused
on.
And a lot of people are saying that rag is dead because of those larger context windows.
And I think it's important to mention that rag is not dead.
It's evolving.
Even with the rise of huge context windows.
And I'll tell you why it's important.
First of all, we have the cost efficiency because feeding a million token document into a LLM is significantly
slower and far more expensive than simply retrieving just the relevant snippet.
With rag.
Another thing why Rag is important is because of precision and noise reduction and Rag filters to only
the most relevant chunks.
So this is drastically reducing hallucinations and positional biases found in long context methods and
retrieval with intelligent reordering.
So this is a strategy of retrieving the documents improves the answer quality while using fewer tokens
than the full context, and this is proven.
I also want to mention that Rag gives us a lot of user facing features, so you can see the source of
the pieces of information in the answer.
So we can trace back that answer to it and find its origins.
And this is critical to convey trust in the AI system.
And it's very critical in regulated environments.
So I think that rather than saying rag is dead, I think that long context models are complementary
to it, and they support Rag more effectively by handling richer and prompt sequences.
All right.
And the TLDR for everything is that larger context windows don't kill rag, but they amplify it and
magnify the strings.
Alrighty, let's go now and take all of these chunks.
We want to turn them into vectors and let's go and index them in the vector store.