Hey there.
Eden here.
And as you know, before we implement an advanced RAC solution, we first need to index our documents
into a vector store.
So in this video we're going to be implementing the ingestion py file where we're going to load articles
into linked documents.
We're going to chunk them up into smaller pieces.
And then we're going to embed them and store it in Chrome ADB open source vector store.
And one quick disclaimer here in a JNI application and specifically in a react based application, and
here we're implementing a very advanced version of RAC.
Then the ingestion pipeline can have a lot optimizations in it.
We can optimize every step of this pipeline from loading the documents when we transform it and chunk
it, and up until we embed it, depending on the model we're using.
And in this project, we're going to be focused on the retrieval part rather than the ingestion.
So in the ingestion part, I'm simply going to go back to the defaults and I'm not going to do anything
special.
Simply chunk up the documents and load them into the vector store.
However, in the retrieval part we're going to be implementing with Landgraf very advanced techniques
for retrieval.
Alrighty, let's go to the code and let's start with our imports.
We want to import load dot env to load the environment variables.
We're going to be using the recursive character text splitter to split up our documents.
We're going to use web based loader to load the documents from the internet.
And we want to use chroma as our vector store.
And let's import OpenAI embeddings for the embeddings module.
Finally, I want to load the environment variables.
And let's create a list of URLs, which is going to be the URLs that we're going to be scraping.
So we're going to be scraping those kinds of Articles about generative AI.
And this one is discussing autonomous agents and all the cool topics of memory and planning and reasoning.
So this is one article.
The other article is about prompt engineering.
So we can see, um, yeah, we can see we have all the prompt engineering techniques zero shot, few
shot, chain of thought, react, etc. and the last one is going to be about adversarial attacks on
LLM security.
So how to hack prompt hacking all of those.
And those are the articles we're going to be ingesting.
All right.
Let's go to the code.
And let's first start by loading the URLs into link chain documents.
So I'm going to use the web based loader.
I'm going to plug in the URLs for each URL.
And now I'm going to have a list of linked chain documents.
So if we'll run this in debug just to examine the output of this, we can evaluate this expression.
And we'll see that what we get back is a list that every element in that list is going to be a list
that contains only one document.
That's the content of that URL loaded into a link chain document.
So we want to flatten that list.
So let's go and do that.
So we're going to iterate through the docs.
And each item here is going to be a sublist.
And each item in that sublist is going to be the document that we want.
So eventually we'll get the document list.
All right so let's debug and check out that we do have a document list.
And we want to evaluate this expression over here.
And we can see we indeed have a list with three elements.
And each one is a length chain document.
Cool.
So we've loaded our content.
Now we want to split it up into chunks.
And now we want to use the recursive character tag splitter.
We'll use the from Tik token encoder.
And we want our chunk size to be 250 with no overlap.
So just some basic text splitting definitions.
And lastly, we want to use the text splitter to split the documents in the document list.
So eventually we'll get from it a list of smaller chunks.
And let's run this in debug and see that we indeed have the documents chunked up.
Let's go and evaluate this expression.
And we can see we have a lot of chunks here, almost 200.
Cool.
So now we're ready to index them into chroma DB which is going to run locally on our machines.
Alrighty.
So we'll use the chroma objects.
It's going to have a from documents method which is going to take the documents which are going to be
the chunks we're going to call our connection name in our index rag chroma.
And we want to use the OpenAI embeddings.
And I think this will default to small embeddings.
Three and now we want to persist our vector store into our disk.
So we're going to add the argument of persist directory.
And I'm going to give it dot slash dot chroma.
So it will persist in that location under the root directory over here after it's done indexing.
And now we want to create a retriever object a Lang chain retriever from that chroma db.
And we'll initialize an object of the class and use the as retriever method in order to turn it into
a retriever.
So we'll be able to perform similarity searches, and we want to load it from the disk.
So we're going to give it the collection name we gave earlier and the persistent directory.
And of course to supply the embeddings function.
All right let's run it.
And let's see that we indeed index our documents and we create this subdirectory.
And we can see now we have the directory with the persistent storage.
Now I'm going to comment this because we don't want to index this.
Every time we run the program.
We simply want to load everything from the disk.
And let's go and let's run it again to see that everything is working and no errors.
And lastly, all of the code is available on the GitHub repository in the branch three ingestion.
So feel free to compare it and to even use this code in the repository.