Hi in here.
And in this video we're going to cover the imports and main classes that we're going to use during the
ingestion phase in our Rag pipeline.
So we'll also define the environment variables we'll be needing.
So all the API keys and all the configuration we need will initialize to really map to really extract
OpenAI embeddings.
And if you want you can use open source embeddings.
And we'll also be using the pinecone vector store.
And you can use Chrome ADB as well, or any other vector store you want.
The beautiful thing about link chain is that we have one single interface for all vector stores and
all embedding models, so the code should be similar.
All right.
So right now we have this boilerplate code which is importing Asyncio and running our main function
here.
So let's run it as a sanity check just to see that everything is working.
So I'm going to click this play button at the top right.
And then everything is going to run now and it's working.
So let me go and show you the logger.py file, which I have pre-prepared and you should have it as well.
And here I simply define a bunch of colors and some nice printing functions, so we can see the logs
in a readable way.
And we can document and log every step of our ingestion pipeline.
So we have log info log success log error log warning log header.
And we'll be using an importing that in the ingestion file.
All right so let's start with the first import.
We'll be needing the OS module so we can import and use in reference environment variables.
We need SSL because we're going to create some SSL context and some type hinting objects.
All right.
Now we want to import the certified package, which is going to get us a valid certificate, so we can
attach to our HTTP request that we're going to be sending.
Like always, we're going to import from dot env the dot env file, which is going to load the environment
variables from the dot env file.
And we'll soon see exactly what values do we need there.
And next we want to import all things related to link chain.
So let me go and paste all those imports and let's go over them one by one.
We're going to import first the recursive character text splitter which is the link chain helper class
which is going to help us split the documents.
Now I have an entire video dedicated discussing how this text splitter is working and why it's effective,
so feel free to check out this video in your free time.
We want to import from link chain chroma.
The chroma class, which is going to be the chroma vector store in case you want to index everything
locally.
However, I will be using the Pinecone Vector store, which is a cloud based vector store, so it's
going to be in the link chain pinecone package.
So you can choose either one.
I'm going to show in the videos the Pinecone Vector store.
I also want to import the document class which represents a text document with associated metadata.
So it's the core abstraction in link chain for handling text data that can be processed, split, embedded
or indexed.
And if you want to dive deep about documents, you should check out this video where I elaborate why
this class is important and why do we need it, and examples of it.
And I'll be using the OpenAI embeddings.
And by the way, I have students that use open source embeddings which worked fine as well.
All right.
Let's go and import from Tivoli.
The Tivoli Crawl, which is going to be our main driver to get the documentation, and we'll also import
the vanilla extract into Vmap for the optional video later.
And that's it for the link chain imports.
And now we want to import from our logger.py file.
We want to import all the logging functions.
Cool.
So we're pretty much done with the imports.
And now I want to load the environment variables from my dot env file.
And let me show you which environment variables I have.
The OpenAI API key for the embeddings, the pinecone API key for the pinecone vector store, and then
the LinkedIn API key with link chain tracing v2 equals true and link chain project with the name documentation
helper.
And those three environment variables is to have Langschmidt trace our pipeline.
And lastly the Nvidia API key.
So we can use the Vmap and extract.
As always never share your API keys.
I'll be revoking those API keys once I finish editing this video.
Cool.
So just like we did in previous video, let me now configure the SSL context with a valid certificate
using the certified package.
And again, I remind you this is for making tons of requests for the API which we need that.
So we don't want to encounter a weird SSL certificate error.
So this is defensive programming.
Oh, and this reminds me if you're using a corporate computer and your corporate computer has a VPN
running, then you might be still getting a certificate error.
So what I recommend you to do is to disable the VPN so you will be able to make those requests.
Cool.
Let's go and initialize the classes that we imported.
So first I want to initialize the OpenAI embeddings class.
I'm going to give it the model of text embedding three small.
And if you want to see a cute progress bar while your text is being indexed into a vector, you can
enable this show progress bar flag.
So this flag is disabled by default.
I am simply writing it explicitly to show you that it exists, and we have the argument of chunk size
equals to 50.
And this is going to limit how many text objects blockchain documents were going to embed at every request
we send to OpenAI.
Now why is this important if this number is going to be too big, for example, a thousand, then this
means we can send a thousand of text objects to be embedded in OpenAI.
And depending on our customer tier in OpenAI, and it's going to be the same for any cloud provider
that is giving us an embeddings model.
We get a token per limit limitation that we can embed.
So here we are limiting the number of documents we can embed per request here, and we are limiting
it to 50.
So it's going to be 50 linkchain documents and text objects that we're going to embed at a single request.
So again this is for rate limiting.
If we put this number too high then we're going to get rate limited.
And if this number is going to get to be too low for example one, then it's going to take a lot longer
to process and to embed everything.
All right.
Let's talk about the retry mean seconds.
So in case a batch of hours fails for some reason maybe something is wrong with the payload.
Maybe we are rate limited.
So a lot of times it's going to be rate limits.
And if it's going to be rate limits there are a bunch of strategies of how to handle them.
So one strategy is to use this retry min seconds.
And this means that after a failure we are going to wait 10s before we try to retry it.
And if this number is going to be very big, for example, 60s a minute, then this means that after
each failure, we're going to wait at least one minute before we're going to retry to embed those texts.
So I want to show you this error over here.
So this is an error from my previous tries of indexing this documentation here.
Now I am running a lot of concurrent batch requests.
So this means I'm running concurrent requests where each requests have a batch of a lot of documents
here.
And you can see that some of my batches are being rate limited.
And you can see that in the error message I get I get here A429.
So this is a common rate limiting error code.
And you can see that the message states how much time do I need to wait in order for the rate limit
to reset and my request to pass through.
So you can see for example, 194 milliseconds, 500 milliseconds, etc..
So if I put a very high value in my minimum weight, it's a heuristic that my requests are going to
pass through and they're going to wait enough time.
So I can go and embed the rest of my documents.
Now of course this is a trade off because if I wait too long, then my entire processing time is going
to be too long as well.
And rate limiting errors and handling rate limiting is very common when you take an application to production
and when you handle scale.
And the rate limiting concept is basically for every third party you're going to use, which is cloud
based.
And it's very known in the industry.
And there are many algorithms of how to calculate rate limiting.
Each vendor calculates rate limiting differently.
You have token bucket, leaky bucket, and tons of algorithms that help you do this.
We're not going to get into this, but overall, you should know and you should understand the concept.
And especially in generative AI applications we need to handle rate limiting.
So here when we enable retry mean seconds equals ten then this is some way of handling this.
All right so let me paste this snippet over here and let me cover the rest of the objects.
Initialization.
What I have here in commented out is initialization of a local ODB vector store.
Now they persist directory is going to be where it's going to be persisted.
Now I marked it chrome underscore db.
So this means that the DB is going to be stored under the current working directory of our project.
And it's going to create a Chrome DB directory.
So you can see I have it right here on the left side.
And I give it the embedding function of the embeddings objects I created earlier.
So this is in case you would want to use chrome db.
However I will be using the pine cone cloud based vector store.
And here I'm giving it the index name of length index 2025.
And let me go to Pine Cone and let me create an index.
Let me call it Length docs 2025.
And in the embeddings model here I want to select OpenAI's embedding text small, and I'll choose the
dimension to be one 536.
I'll go with the serverless option because I don't want to handle any scaling and all those abilities
myself.
And let me go with the default cloud provider here, which is going to be AWS.
And I'll go also with the default region.
Let me go and click here create.
And my index is live and running.
And we can use it now right.
Let's go back to the code.
And I remind you, when we created the index, then we gave it the dimensions that are going to match
the text embeddings three small embedding size.
And here I'm initializing all the objects with the emphasis on which is going to take the URL and get
a documentation from it.
And also to extract and to map for alternative way of scraping and crawling a bit more manually the
link and documentation.
But this is going to be on the optional video.
Ah wow, this video was pretty long.
We are done with all the initialization.
Let's go and run everything just to see that everything is compiles and we don't get any initialization
error.
And we can see it ran successfully and we didn't get any error.
Amazing.