What are splitters in long chain and why it is important for you to master this technique.
So, splitters.
Uh, we use the splitters to split documents in the rack technique.
Remember, the rack technique is one central technique.
In order to, uh, uh, be able to use the LM models with your own private data.
Right.
So, for example, to be able to ask ChatGPT about private data from your company or about a book or
about a private database or whatever, in order to do that.
Since the LM models are limited by the context window, we use a technique called the rack technique.
And you remember the rack technique.
What it does is it takes this huge private document you want to combine with, with ChatGPT for example,
and it breaks it down in small chunks of text.
So after breaking breaking it down in small chunks of text, it converts the chunks of text into numbers.
And then it loads the these numbers into a vector database.
And there is where you can start playing with data.
So you remember the steps we follow in this rack technique are.
First, we use splitters to create chunks of text.
Second, we use embeddings to convert the chunks of text into numbers.
And then we load it into a vector database.
So the splitters are the first part of the rack technique.
And using them correctly can very positively affect the quality of our rack application.
So it's very important to master issues such as the technique for creating relevant text chunks, or
the technique for adding relevant metadata to a text chunks.
And this in the example.
So we are going to see a few examples of of splitters.
The most important thing for me is that you get the concept that you can start using a working code,
you know, to experiment, to try, etc. and you understand where to find more information.
Right?
So we have not going into the small details because that's why you have the working code for you to
experiment.
But the we are going to stop in the concept and in the, in the main ideas.
So I just, uh, I want to tell you, uh, why it is very important for us to master techniques like
this here.
So if we have a document, this is a sample document that, uh, we want to use with our rack application.
Right.
So it is going to be very different if we divide this document into one word, for example, fragments
or one word chunks or three word chunks or paragraph chunks.
So if we separate this document into paragraphs or into whole sentences, ChatGPT is going to be a much
more proficient, uh, the quality of the application that our application is going to be much better
than if we break this document into words.
Why?
Because for ChatGPT and for LM models, uh, it is very important the context.
What why do I mean with the word context here?
So if you use, for example, the word red.
A.
Probably a person or ChatGPT.
They don't know what this read is about.
Are you talking about a dress?
The color of a dress?
Are you talking about a light, a traffic light?
Are you talking about the politics?
What are you talking about when you say red?
So the context around a word is important.
So that's why for ChatGPT is much more useful if we give a sentence than if we give one separate word
or three separate words or words altogether.
Right.
So that's why it is important to master the techniques around splitters.
Right.
So you will see here a few ways of doing things.
Some of them are good, some of them are not.
And you will see that some of them fail.
Right.
So for example, the first thing we are going to use here is character splitter is the more simple splitter
long chain provide us.
And here you see that.
Well we are telling, you know, the size of the chunk and the size of the over the chunk overlap we
want to have, etc..
But you see that this splitter is not very good because when we provide, you know, a document which
is longer than 26 characters, it doesn't divide it, you know, or even a longer document, it doesn't
divide this.
So this splitter is not good.
But if we take a look at the recursive character splitter, which is the most commonly used, the most
popular in lunch you will see different results.
So applying a recursive character text splitter, we immediately see how to separate the first text.
And you will see the overlap here.
So the termination the ending of this initial chunk is the beginning of the second chunk.
And you can see how this recursive splitter apply to the long document works.
So we have a lot of, you know, chunks of text.
But you will see that even if this chunk of text look okay in a similar length, etc., etc., you will
see that they are not going to be very useful for ChatGPT, or at least they are going to be less useful
than if we have just a couple of chunks of text, one paragraph and then the other, because this way
is going to give much more, uh, much more context to ChatGPT.
So he's going to be much more proficient in order to respond to our questions.
So that's the things that that's the kind of things, you know, that we are going to learn in the next,
uh, chapters.
Another interesting concept is, uh, that adding helpful metadata to the text chunks is going to be
good for the performance of the rack application.
So, uh, think of, uh, this same text.
So let's say that instead of having two paragraphs, you have like 1 million paragraphs or, you know,
10,000 or whatever, right?
And you load paragraph by paragraph into the vector database, okay.
It's better than word by word or three words length or whatever.
But still it can be improved with some metadata added, added to, uh, each of the chunks of text.
For example, if we add to this chunk of text some metadata like the name of the document, the page
of the document, the I don't know the section of the document or the year of the document.
It can be.
It can be very helpful for ChatGPT in order to search, you know and understand what is the response
that it has to provide.
So a little example about that is the last exercise.
So in the last exercise we are making use of a module called markdown header text splitter.
And what this module does is it uses the markdown as metadata.
So probably you are familiar with markdown.
If you are not familiar with it, just Google it or go to ChatGPT and and ask, you know what is markdown?
You will see that markdown is a way to write text to provide format.
So for example, here we are using markdown.
In order to make these letters a title.
We could use markdown in order to make these letters cursive or whatever.
Right.
So these symbols you see here are markdown symbols.
These for example is is telling that this thing my book or title, my book is a title format.
And this second thing is this is smaller in size and this is smaller in.
Size, etc., etc..
Right.
So what this module does is, is it is going to use this markdown information in order to uh, include
metadata, uh, with our chunks of text.
And I think the easier way to understand this is to see one chunk of text.
So here you have how this splitter A builds a chunk of text.
So this is the chunk of text I was born in a very sunny day of summer.
I was born in a very sunny day of summer, but.
With this chunk of text, the splitter is providing a metadata.
So this metadata is telling us this chunk of text belongs to a book with this title, to the chapter,
one of this title that has this particular name.
So I was born in a very sunny day of summer.
Of summer belongs to the chapter one of the book called My book.
Okay, so.
That's it for this chapter.
The most important thing you need to to get from this chapter is that splitters are the first step of
the rack technique, and that the quality of your rack applications is going to be affected by how correctly
you use a splitter.
So it is very important to master this technique.
And it's also very important to master the embedding technique for example.
So we will study this further.
But you right now can start, you know, experimenting with the text, going to the chain documentation
to know more about that and to experiment with solutions that can respond to your particular cases.