Instructor: LLMs, regardless
of their specific architecture,
all have predefined token limits.
So this will determine the maximum number
of tokens they can process in a single interaction.
Nowadays, most LLMs have the token limit
of around 4K tokens,
but the number of tokens that the LLM would be able
to digest keeps growing and growing over time.
So we even saw right now that Anthropic
came out with a model that can't ingest 100K tokens,
which is a dramatic improvement
to what's currently on the market.
Anyways, even though the number of tokens
of the LLMs will continue
to grow and grow, we still will have a limit
that we will need to get by and to find the workaround.
Now, for simplicity, let's assume
that each token is equal to a one word.
Now, the total count of tokens
will include both of the input prompt
and the generated response that the LLM gives us.
So as long as both
of them do not surpass the token limit, 4K, in this example,
then we will be fine and everything will work.
Now, in an LLM interaction, the LLM does not care
how do we divide our token limit?
Now, we can create a prompt with very few tokens
and request LLM to return us an elaborate response
so that way, most of the tokens
are being spent on the response part,
but we can divide it half and half
and we can even say that we want our huge prompt
and we want a very quick and concise response.
The LLM won't really care.
As long as we don't surpass this token limit,
it would be able to digest it and work fine.
But in real world applications
and in advanced usage of LLM,
we will hit the token limit for sure.
This can happen for a variety of reasons,
like we're having a prompt which its context is too large
and really, this problem is inevitable.
So once we surpass the token limit,
we will get an error from the LLM stating
that we can't use it because we sent it too many tokens.
Now, there are a couple of strategies
to solve this token limitation.
A LangChain supports all of them.
The first one is called stuffing.
We have map reduce, refine,
we're going to discuss in this session.
Now, the best way to explain those strategies
is by using a real world example
of the problem of summarization.
So let's say we have a bunch of documents
that we want to summarize.
So we can use the load_summarize_chain.
Now, this function will give us a LangChain chain
that summarize documents.
Now, when we initialize this chain,
we can pass it the argument of the chain type.
When chain type is equal to stuff,
we are telling LangChain
that we want our summarization chain to handle the context
with the stuffing strategy.
Now, what stuffing?
So exactly like a stuffed animal doll
is stuffed with cotton and with fiber,
we are telling LangChain that we want to stuff our context,
which is the documents into the prompt as is.
So we're simply pushing everything into the prompt
as is without doing any alterations.
Now, this will cost us one API call to the LLM.
Now, maybe this is the most intuitive thing to do,
but eventually if we will use more than a couple
of documents, we'll hit the token limit
because if we have a lot of documents,
then we'll for sure have a lot of tokens.
Now, even if we take an edge case
and let's hypothetically assume that there is an LLM model
that can ingest any size of tokens,
then we will still encounter the barrier of the payload
we can send the server which would process that request.
Okay, so one way to solve this
is to use the map_reduce chain.
Now, this time we're using the load_summarize_chain,
but we pass it the chain type of map_reduce.
Now, in this chain,
what we're doing is taking all the documents
that we use to stuff in our prompt
and send directly to the LLM.
And from each document we're going to be making
a new prompt which will hold the instructions to summary
and a context, which will be the document.
Basically what we did comes from functional programming
and it's called mapping in the formal terminology.
And it's basically applying a transformation function
to a collection
of some things, like in this example, documents
and to produce from it a new collection.
The new collection is going to be a collection
of prompts with the context.
And now, we want to create a new mapping step,
which will take the prompts
that we created from the documents
and each one of those prompts will send to the LLM.
So this transformation will take the prompts,
make an API call to the LLM
and get the summary for each document.
Now, notice that everything can run in parallel over here,
and that's a very great advantage
that will optimize performance and we can do that
because the documents are not dependent on each other
and can be standalone.
Now, the last step would be to take
all those small summaries
that we had made from the original documents
and to create a big summary.
That is going to be the summary
of all those summaries we have prepared.
Now, in the professional terminology,
this is called reducing
and it's basically applying a reduction function.
This one would be to call the LLM to an interval
and to produce a single value.
Now, the single value here is going to be the final salary.
And how does this entire process looks in LangChain?
Well, it's actually one line.
So all we need to do in order to perform all this
is to simply select the chain time to be of map_reduce.
Now, this is very, very cool
and this is where LangChain shines
because it does all the heavy lifting for us
and all of this entire
and lengthy process, it does it for us.
Now, the advantages of using map_reduce
is that it can scale to an enormous number of documents
and it can also run in parallel.
So we'll have optimized performance and it'll run fast.
However, it does come with some disadvantages
because we're going to be making a lot of API calls
and this may impact costs
and we might lose informations in the mapping process
where we summarize each document.
So we might be losing some context.
And basically, this is prone to be losing information.
Okay, the next way to handle this is using the refine chain.
Now, I think the best way to explain
what the refine chain does is
to talk about a concept from functional programming,
and this is the function foldl.
I'm going to explain this function,
which seems to be unrelated,
and then we can talk about the refinement strategy
and you'll see how those topics connect to each other.
So in functional programming, the foldl function
or fold left is what is called a higher order function
that is commonly used for iterating over a list
and accumulating a result by applying a binary function
for each element and accumulating the value so far.
Now, a binary function
or a binary operation is simply a fancy word of saying
that this is a function that needs to receive two arguments.
So what the foldl function does is apply the binary function
to the initial value
and the first element of the list.
This will produce a new accumulated value.
Then it applies the binary function again
to the new accumulated value and the second element
and so on and so on until it reach the end of the list,
and we manage to reduce the list into a single value.
So let's revisit this
and let's explain the example we see right now.
The function foldl or fold left has three parameters.
The first one is the function to apply.
The second one is the initial value,
and the third parameter is the list
to apply the function to.
Now, let's explain how does foldl work.
So we first take the initial value
and we take the function to apply.
Now, we simply take the first element from the list
and then we call the function
with those two arguments.
So in this example, we have the initial value one.
We take it and multiply it with the first element
of the list, which is one, and the result is one.
And now, we continue to do it with the rest of the list.
So we have the result, which is one,
and we then take the second element
from the list, which is two.
We now use the function to apply
and we perform one times two,
and then we get the result of two.
Now, we continue this, so we take the result two,
we take now the third element for the list, which is three,
and we take the function to apply
and we perform two times three, which is now six.
This is the result.
Now we are left with the last element of the list.
So we take that element
and we call the function to apply six times four,
and eventually, we reduce this entire list into 24
using the initial value and function to apply.
Now, I want you to imagine that instead of the function
to apply to be the multiplication operator,
going to provide a function that will receive two documents
and it's going to combine those documents
and summarize them.
And the initial value, I'm not going to provide one,
but I'm simply going to provide an empty string
or a document that represent an empty document.
And in the list parameter, I'm going to provide the list
of documents that we want to summarize.
Now, I want you to imagine
what would happen if we'll take the function foldl
and apply it to those arguments that we just mentioned.
So basically, we'll summarize an empty document
and the first document into a first summary.
We'll take this first summary
and then we'll take this second document,
combine them together
and create another refined summarization.
So we keep refining
and refining the summarization until we end up
with the perfect summary for all of the documents.
So this is basically the implementation of the refine chain.
So it's very, very cool
and LangChain is doing all of this background work
and heavy lifting for us, which is super, super cool
and save us a lot of time.
Now, in my opinion, this is simply mind blowing
of how elegant and smart this framework is.