-: Now that you've explored a range
of different summarization techniques,
let's move on to how can we load in documents
and text splitting.
We've got an example here using another document loader
and we have this package called Beautiful Soup.
So you'll need to install Beautiful Soup when using this.
We then import requests
and we've got this Read Me file
from a GitHub repository.
We make a get URL to get the Read Me.
We then get the text
and we save that locally in a Read Me dot MD file,
which we then load in.
So if I run all these bits of code,
what you'll see is we generate these docs
and these documents are Lang chain documents.
You can also get these Lang chain documents directly
by just typing in Lang chain dot schema import document,
and you can create documents just like this.
So we can have a document here
and we can define its page content.
You can also add something called metadata.
So that is how you can make documents inside Python
and you'll generally have a list
of these Lang chain documents.
Now these loaders also provide us a nice way
for us to be able to automatically get Lang chain documents
directly from different file sources like Markdown,
GitHub, PDFs, any kind of source you can think of,
notion databases.
What we need to do though, is we need to make sure
that the documents are not too large.
And to do that we can use a text splitter.
So you'll see this document content is quite large.
It might not necessarily all fit into
the large language model.
So what we can do is use something called a
recursive text splitter.
So what we can do is use something called
a recursive character text splitter,
which will split based on a series
of characters recursively,
making sure that the splits
adhere to specific things.
And those things are the chunk size,
the amount of overlap,
the length function,
and is it gonna be separated or not.
So let's show you a really, a chunk size of 300.
And what we're doing here is we make our text splitter,
then we take our documents in that we load
from the Read Me file,
we can then split those documents into chunks.
And so you'll now see that we've got all these different
documents that have got chunks,
and we can also increase the overlap,
which will mean that we have more documents, right?
So if I then look at the length of these final documents,
you can see I have 182.
As soon as I start reducing the overlap
for the chunk size,
we then start getting less and less documents, right?
Because the overlap will cause the window chunks,
you have different amounts of overlap, right?
And therefore what we're doing really is
as we increase the overlap,
there's a larger number of chunks that we need to create.
However, we are less likely to lose information.
I find that, you know, a small overlap is good.
And then what we can do is we can make a model.
So using the chat, Open AI,
loading on our load summarize chain,
and then we could do a chain type of MapReduce
on the input documents that we had.
And I'm not gonna necessarily show this
'cause it will take a while to run.
But essentially what we're doing here
is we are using our input docs that we got from here
and then we then put that as the variable in our input docs.
And this will use a MapReduce summary.
So just what we learned in the previous lesson.
It'll use a MapReduce specific type of summary
to create individual summaries on each document.
And then what it will do is
after it's made the summaries,
it will then summarize those summaries
inside of a reduced step.
Cool, I'll see you all in the next one.