In this section, we'll dive into the technique which is called retrieval augmentation generation,
also known as Rag.
But before we go and dive into the technical implementation, I want to discuss the motivation for this
technique and what this technique is going to help us achieve.
And what is it solving.
All right.
So let's take the use case that we have a very large document.
And this document has tons of information inside it.
So it may be maybe hundreds of pages.
And in this example I took the book Harry Potter and the Sorcerer's Stone, which is a pretty long book.
Now, we might want to ask the LLM questions on this book, and specifically maybe questions focused
on different areas in the book, or maybe specific scenes, maybe specific paragraphs.
So questions, for example, like how to make a certain type of potion where the answer resides in a
very particular paragraph.
This example illustrates a problem regarding to Harry Potter, but it also applies to other fields.
For example, if we have a very large financial document that we want to ask question about, and we
want to find certain clause in that document, and that's where the answer is going to come from.
And this problem is especially important when we're dealing with private data, because large language
models are not trained on these private data, so they are not aware of it.
So because our financial document is private, the LLM doesn't really know anything about it.
So we need to find a way for the LLM to effectively question answer over this document.
So this is the problem that we're solving here.
Now let's talk about the solutions.
Now a very naive solution is going to take the entire book.
Let's say it's a PDF document.
And to simply go and plug everything into our prompt that we sent LN.
So we're going to stuff the LM with the entire book and we are going to ask the question.
So here's the placeholder for the user's question.
And here's a placeholder.
Where are we going to plug the entire book of Harry Potter here.
Now this solution might work some of the times, but it has a lot of problems with it and it doesn't
really scale.
There is an inherent hard limit of how much text we can feed the LM, so if the document is going to
be too long and if this book is going to be too long, we're not going to be able to fit everything
because we're going to exceed the LMS token limits.
And even if we have LMS with 1 million or 2 million token limit, which is quite common these days,
this solution is still not ideal.
It has many other disadvantages.
For example, the LM is going to be much less effective with very long prompts and very long context.
And these have been proven in research.
For example, the needle in the haystack research, which clearly shows that large language models,
even with huge token limits, they get less effective with very long prompts.
There is also the issue for cost, because larger prompts are going to cost us more, and there is a
latency issue because larger prompts are going to take longer to process.
So let's conclude that we have four problems here.
The first problem is that we have a very hard limit.
What we can put in, and is that we have the needle in the haystack problem, which means that the longer
the prompt, the less effective the answer from the LM.
Third, we have the cost issue.
And for and last we have the latency issue here.
All right.
So let's look now on another solution here.
So the second solution is going to require us to add some pre-processing.
And it's going to require us to take the original document.
And doesn't matter how long it is, we're going to be splitting it into smaller chunks.
Now these chunking process can be naive and it can also be complex.
We have a long range of how to do it.
And we're going to be discussing this in the course.
So let's assume now that we have those chunks now ready.
Now in the second solution, instead of plugging in the entire book, we're going to add another step,
which is going to take the user's query, and it's going to find the most relevant chunk for that query.
We're going to plug to the LM call only that relevant chunk, which is the most relevant to the question
here.
So now we are focusing the large language model to answer and to ground the answer only on something
which is the relevant piece of data that is going to answer the question.
So the LLM is going to have a much easier time of answering our question.
So instead of sending the entire book, we are going to send only a specific paragraph or only a couple
of paragraphs here.
And we are going to solve all of the problems we talked about.
We are not going to pass the hard token limit of the LLM because we are sending much smaller piece of
context.
We won't encounter the needle in the haystack problem because we are only sending very, very specific
pieces of text which are the most relevant for the question going to cost less because we're sending
fewer tokens to the large language model.
And of course, the processing time of the LLM is going to be faster because it's going to process less
tokens.
This technique can scale to very large documents, and it can even work with multiple documents here.
Now, it does have its drawbacks.
So it's going to require us to add a pre-processing step to chunk the large documents.
There is a lot of depth into this chunking mechanism.
How do we chunk the document?
What are going to be the tokens that we split the documents from?
How are we going to make sure that each chunk is going to have the relevant data?
And what if we're dealing not with the document but with a code repository?
We need different chunking strategies to different type of documents.
And what if we do not know what is the content of the document.
And we get it dynamically for the user.
So there is a lot of depth in this pre-processing part.
And we are going to be covering this in the course.
Now another downside is that we need to have some kind of searching mechanism to find those relevant
chunks.
And what if those relevant chunks are not that relevant, and we need additional context to send to
the LM to answer the user's query.
In this section, we're going to be answering all of those challenges.
And by the way solution number two that we discussed this is actually rack retrieval augmentation generation.
And what we actually saw.
It was a very high level overview of rack.
Now in rack retrieval is for retrieving the relevant chunks.
Augmentation means that we take our prompt and we augment it with those relevant chunks.
And generation means to simply send to the LM and to use that when we make the query to the LM.
So what we saw in this video is really the motivation and the intuition for what's rack.
So let's go and let's now learn about the implementation itself and learn how to implement this kind
of technique.