Teacher: A lot of LLM applications involve
somewhat connecting LLMs to external data sources.
Like we saw in the course,
where we took Medium articles
and we talked the link chain documentation
and we QAed over them
by using the retrieval augmentation generation method.
But a prerequisite for doing it is to ingest the data
into a format that the LLM can easily understand it.
So, most of the time it'll mean
that we need to take our data
and ingest it into a vector store
like we showed in the course.
But before we do it, we need to first split it
or chunkify it into smaller chunks
because we can't just put all of the data as is
in the vector store.
We need to have smaller chunks
so we won't pass the token limitation.
So, while this task may seem trivial,
and we saw this in the course, it's often nuanced
and overlooked because when splitting the text,
we want to ensure that each chunk
has some cohesive information that we can understand
and the LLM can understand.
We don't want to simply split a sentence
in the middle of it.
So, we want to have a chunk that is small enough
that we can understand it,
but not too big
and not too large,
so that it will have a lot of tokens.
So, I often get messages of questions
of how to split the data, how to set the chunk overlap,
and what should be the chunk size, or what should we split.
And to be honest, there is no correct answer for this,
and every case needs to be examined
and need to be handled differently.
However, luckily for us, lang chain created a tool
that we can use to visualize how we split our text.
So, this is called the Text Splitter Playground.
So, let me show how to find it.
I'm simply going to Google lang chain
text splitting playground,
and I'm going to select the second result.
This is the Git Hub repo. It's open source.
You can check out this code.
And this is basically a streaming publication.
You can click on this link
because it's hosted on lang chain.
It's very intuitive to use.
We can see at the first part,
we can play around with the parameters.
So, we can play around with the chunk size,
with the chunk overlap,
and how do we calculate this chunk size.
And we can even select our text splitter.
So, right now we're using
the most common recursive character text splitter
that we used in the course.
And at the bottom, each time we change this parameter,
we can see it reflected in the code,
which we can simply copy paste into our workspace.
Let's see this in action.
And let me go to the lang chain blog
and select a blog that we can copy data from.
And let's see how we chunk it up.
So, I'm simply going to copy all the text
and I'm going to go back into the text splitter, pasting it.
Let's click split text
and let's take a look at the chunks.
We can see visually all the chunks
and we can make if they make sense or not.
This way we can optimize our chunking strategy
and to actually visualize and see how our chunks looks.
And those are the chunks which are going
to be embedded at the vector store, for example.
This is also a good way
to check out the chunk overlap.
So, for example, we can see between two chunks,
what is overlapped according to the overlapping size.
And to wrap up this video,
I really think this is a wonderful tool,
which is helpful when we want
to optimize our chunking strategy
and when we want to visualize our chunks
and what data they hold.