The recursive character splitter.
Wow.
It was hard to pronounce.
It's a utility class which helps us to split large documents into smaller chunks.
How does it do it?
It does that recursively by characters.
Recursive aspect of this splitter refers to the strategy of splitting the text by progressively trying
different levels of separators in a hierarchical order, until the chunks are small enough.
So it starts by attempting to split the text using a larger separators that correspond to bigger semantic
units.
So, for example, paragraphs are going to be splitted by double newlines backslash backslash n.
So for example, we want to start breaking the text into chunks with the separator.
So after we do that, if a chunk is still too large, even after the split, it will recursively remove
other smaller separators like a single newline.
So this is to represent the sentence a backslash n, and then we can even go to spaces which are going
to separate words.
And finally down to the individual character if necessary.
And this recursive approach tries to keep semantically related text together as much as possible.
So this to preserve the natural language flow and coherence within the chunks.
Of course, this is a heuristic, and this doesn't always mean that it will be split it up coherently.
And this method actually contrasts the simple fixed length splitting by characters or by token, because
it respects the inherent structure of text to maintain semantic integrity within chunks.
So it's a different strategy.