Eden: Hey, there.
Eden here, and quick update before we dive in.
So you might notice in this video that I'm going
to be using PyCharm, and you'll see a peep file
and peep file.log in the project.
And that's because in this particular video was recorded
in earlier versions of the course.
Now, I've gone through
and rerecorded almost every single video
to keep everything up to date with the latest versions
of LangChains and the other libraries
we're going to be using.
However, in this specific lesson,
the code is exactly the same.
It hasn't changed at all.
So rather than rerecord something that's identical,
I kept this one as is.
Now, that being said, if this really bothers you,
just let me know and drop a comment or send me a message.
This video is on my list to rerecord,
but I've got a lot of other updates
and new content I'm working on, so I'm prioritizing things
that actually affect your learning.
All right, so now that is out of the way.
Let's get into it.
Let's review the file that we're going to ingest.
So this is the medium block.
Now, let's initialize the text loader.
So I'm going to create a text loader object
and I'm going to give it the path of the file we want
to load, and you need to change this path to match the path
where it's saved in your file system.
And then all we need
to do is simply use the method loader.load.
Now, this would load the file into a LangChain document.
Now, the beautiful thing about this abstraction
is that if we were to use WhatsApp
for loading WhatsApp messages or Notion for Notion Notebooks
or Google Drive files, then the interface would be the same.
Would need to create the document loader specifically
for that third party
and we would need to use the load method.
I just want to show you how rich this ecosystem is.
If we'll head up to the LangChain documentation
into the document loader section, we can see
that on the left side we have some document loaders
that LangChain implemented.
So those are for generic file types like CSV,
HTML, JSON, PDF.
But if we go to integration,
and this is a part of LangChain community
because everyone from the community can upload
a third party loader, then we'll see that we have tons
of document loaders.
Now, let's take an example, and let's say that we want
to load some YouTube transcripts.
So the interface is going to be the same
to simply use the loader.load.
If we wanted to load Slack messages,
we can use the Slack loader,
and use the loader.load.
So this is a very nice abstraction in my opinion,
and you can really see the power of the community
with the amount of loaders that we have.
Cool.
Co we've loaded the document,
so now it's time to split it into chunks.
But before that, let's just run everything in debug
and let's examine all the objects.
But before we begin, I just want to note that some of you,
depending where you live
and what kind of encoder your operating system uses,
you might get an error.
So this error is going to be looking like this.
So it's going to have the string
of unicode error in codec can't decode bytes.
Here is another example of this error.
And luckily for us, fixing it is very, very easy.
You need to add the flag of encoding
equals to the string of UTF-8.
And if that doesn't work for you, you can use the flag
of auto detect encoding equals true.
So one of these flags should definitely fix your issue.
However, for most people,
everything should be fine even without those flags
like for me in the video.
So we can examine the text loader object
and we can have a look in all the metadata
we initialize this object with, like the file path,
and we can use different encoding systems if you want to.
After we invoke the loader.load method,
then we get back a list of LangChain documents.
So let's examine the document variable,
and we can see it contains a list of only one document.
And all the elements
of that list are documents of LangChain.
Now, every document has a couple of key attributes.
So one of them is page contact.
So it's all the content that we loaded.
We also have the metadata field,
and this field is going to hold by default the source
of this document.
So we can see now the sources the file path.
And this is used when we use reg to say
where we got that information for,
and where did we ground the LMS answer with.
So it is super important.
And we can add to this metadata field any key
and value that we want.
And this could be used for filtering later
or for segregation of data.
And this is very useful when we deploy things to production
and we implement more advanced reg systems.
All righty.
Let's go and split some text,
and we're going to create a character text splitter object.
And the character text splitter object is something
which can be very complex.
It can use regular expressions,
it can use a different length function to count the number
of tokens that we're splitting it for,
and it has a lot of customizations.
But in this example, we just want
to show you the principle of how it works.
So we're just going to populate two arguments.
Chunk size, we're going to set for 1,000.
So this will limit our chunk size
to 1,000 characters.
And why did I choose 1,000 for the chunk size?
So this is a heuristic.
And a rule thumb when chunking
is to keep the chunk size small enough
so it would fit in the context window.
And the context window usually is going
to be holding a couple of chunks
because we're going to retrieve a couple
of chunks from the documents, and it should be big enough.
So if we would read it as human beings, we would know
what these chunk means, and it has a value
and semantic meaning.
So if the chunk size would be too small,
we simply won't understand anything from it
and it won't help the LLM give us the answer that we want.
So this is a rule of thumb.
And here in this example I've used 1,000.
And splitting text to chunks
and document to chunks is still super important,
even when we have models like Gemini,
which can ingest 1 million tokens,
because in LLMs there is a very nice another rule of thumb,
which means garbage in, garbage out.
So if we send the LLMA lot of information,
which is not relevant, so first thing it's going
to cost us money because the more tokens we send,
the more tokens the LLM digest, then it costs us more money.
And second, it's proven that we'll get worse results.
So if we just use the relevant context
and the relevant chunks,
then we will get better results from the LLM.
Now, let's talk about trunk overlap, and I've put here zero.
Now, this means that all of my chunks are not going
to have overlapping data.
So when is overlapping data useful?
So it's when we don't want to use context between chunks.
So sometimes that will help as well.
But again, this is a simple example.
So we just want to keep things simple
and to give those values.
Let's create the chunks from our medium blog.
So we want to invoke the function,
the method split documents,
which receives a list of documents.
So the variable document is actually holding a list
of LangChain documents
with one document.
Let's debug and examine all the LangChain objects.
So we'll first start by examining the text splitter,
and we can see all the fields that it has,
and it has the chunk size, chunk overlap,
and it also has a separator.
So on what it's going to split the text.
So here is Backslash and Backslash.
And now let's go and examine texts.
So those are the chunks.
Now notice, chunks are still documents.
So those are documents,
but if we'll take a look, for example, about the value
of the of those documents, we can see it's limited here.
So it's much smaller, and here is for 1,000 characters.
And we can also have the metadata of the document.
So what is the source that it came from?
Now, remember the rule of thumb
is that you should read the content of the chunks
and it should make sense and has a semantic value.
And we got a system message from LangChain indicating
that some of those chunks
are having more than 1,000 characters,
more than the token limit.
So this is because we were splitting on Backslash + N
and it's not exact science,
so we might get bigger chunks.
But again, those numbers are pretty small,
and in today's context windows of LLMs
of 32K, 100K, 1 million tokens,
then this is not as much as important
as it used to be in the past.
And at the end we can see we created 20 chunks.
All right.
So now it's time to ingest everything.
We'll initialize an OpenAI embeddings object
and give it the open API key
that we get from our environment variable.
And this will give us the embeddings object.
Now, which embeddings we're going to be using of OpenAI?
So the default one is ada-002,
and we can change it if we want, for example,
multimodal embeddings or if we want other embeddings.
And this object will under the hood
will create an OpenAI client
and it's going to use the OpenAI's API
in order to embed our documents.
Let's add a print that we're going to ingest all
of those chunks,
and now let's ingest them.
And the Pinecone Vector stores has a from documents method.
And not only the Pinecone Vector stores,
but any LangChain vector store.
And it's going to receive text, which is a list
of documents, the embeddings object,
which have all the information
of the embeddings model do we want to use,
and the index name
that we have available in our environment variable.
And what LangChain is going to do, it's going to iterate
through all of the documents, all of the chunks, it's going
to embed each and every one of them,
and then it's going to store it in the vector store.
Now, can we write it ourselves this logic?
It's not that complicated.
So the answer is yes, we can.
But what if later we need to switch
between embeddings models
and maybe even to switch a vector store?
Line Chain offers one single interface that is going
to help us with this task
of ingesting documents into vector stores.
So it give us a lot of flexibility of moving between models
and finding the best one for us to use.
Another reason is that LangChain also implements
using threading, using a Async I/O
in order to run things concurrently
and to handle rate limits in case we get one of them.
So it has a lot of boilerplate support
for a production usage.
And this is the LangChains implementation,
in case you were wondering.
Let's go to add texts.
And we can see over here the logic that we simply iterate
through the text, create the embeddings,
and then we insert them into the vector store.
So we can do this in batches,
we can run this asynchronously.
And this is pretty much it.
It's pretty straightforward.
A lot of code
that we would've needed to write ourselves.
And this is also supported by all vector stores.
Cool.
We're almost finished.
We just need to add another print line
that we finished ingesting the documents.
And now let's run everything.
So I just want to show you that my index is empty,
so we can see medium blogs,
embeddings index is empty at the moment.
And let's run our ingestion file.
Boom, we're done.
Let's go to Pinecone.
Let's refresh the user interface.
And we can see at the right side over here
that we have ingested 20 vectors.
Let's have a look on the data structure
that is stored in Pinecone.
So it has the field of text, which is going
to be our page content of our LangChain document.
And this is the chunk content.
It also has the sources,
and this is going to be our path of our chunk,
and this is to give us proof
of where did our grounding came from?
And of course we have the vector,
which is a list of numbers.
And to summarize this video, we use LangChain
to load our documents with the text loaders.
We split them with the tech splitter into smaller chunks.
We then took the embeddings model
and LangChain help us embed all of those chunks
and store them into the vector store.
So this was the ingestion part
to populate our vector database,
and this is the first part of Reg.
And the second part is the retrieval.
So it's taking the user's question, embedding it,
turning it into a vector, then using the vector store
to find those relevant vectors,
which are the relevant documents,
and this is the relevant context.
Then taking the original question, augmenting it
with the chunks, and then sending everything to the LLM
to get back the answer we want when it's grounded
with relevant context.