Let's go to the top of this file here, and let's create another function.
And let me go and paste this snippet over here for starts.
And we want to define a coroutine.
So we'll call it index documents async.
And the input is going to be a list of documents link chain documents.
It's going to receive an integer of batch size.
And this coroutine is going to take all the documents.
And it's going to batch index them into the vector store that we're going to be using.
So we'll log that.
This is the vector storage phase.
And we'll log the how many documents we are now going to index.
We want to create a variable called batches, which is going to be a list that contains lists of documents.
And we're going to split the list into batches with the batch size.
So this is what this line is going to do here.
After that we want to log how many batches we have.
So let's go and do that.
And let me go and paste here another coroutine which is inside this coroutine which is called ad batch.
And this coroutine is going to receive a batch which is a list of documents.
So you can probably expect that each batch we have from above, we're going to be calling this function
with.
Anyways.
The second argument here is going to be the batch number.
And we need this batch number for logging.
So in case one batch fails we'll know exactly which batch fails.
And what was the issue.
Maybe it had some um non valid documents or any other reason.
And in this function we want to take the vector store.
And here I'm going to be using pinecone.
And we want to use and call in the way this coroutine a ad.
And we want to call it with this batch.
And this is going to be implemented by LinkedIn, and it's going to take all the documents.
It's going to use the embeddings model and transform each document into a vector with the embeddings
model.
And then it's going to index it into the vector store.
And after we do that we want to log success in case we didn't get an exception.
And if we get an exception we want to simply log it and we want to return false.
And if everything succeeded we want to return true.
So we managed to add the batches.
And we're going to process this add batches functions concurrently.
So this is what we're going to do here.
We want to create a variable of tasks.
And here we're going to enumerate over the batches.
And we're going to get a number for every batch here.
And once we have the batch and the batch number we want to create for each batch, a coroutine of the
add batch which is going to receive the batch and the batch number.
So once we do that, tasks is going to hold a list of coroutines of all the batches that we need to
index and we want to process.
And once we do that we can call async IO gather with the tasks.
And we're going to await that.
And that's going to fire up all the coroutines concurrently.
And they are going to run independently.
And they're going to index all the documents into our vector store.
So this is going to optimize our runtime.
The results variable over here is going to hold a list of booleans.
And hopefully all of them are going to be true.
If all of them succeeded.
And if we add some failure it's going to hold false.
And let me go and paste this snippet over here.
And here we simply want to count how many successful batches we had.
And we want to make sure that the number of batches that were successful is equals to the number of
batches.
If so, we'll log that there is a success.
If not, we're simply going to log a warning.
And let's go here to the main function here.
And let's await index documents async.
And let's call it with all the splitted documents.
And we'll give it the batch size of 500.
Now remember depending on which vector store we are going to be using, we will need to adjust this
number here and this batch size here.
It's not a magic number.
We need to find the sweet spot of making this number big enough but not too big.
Because if it's going to be too big, we're going to be rate limited and we can be rate limited from
the embeddings model.
So the embeddings model have a tokens per minute or tokens per second, a limit that we do not want
to go through.
And we can even have a vector store limitation.
So the vector store which is going to be cloud based, usually has also a limit of how much it can process
per minute or per second, but usually we do not hit these limits.
The main limitation here is going to be our embeddings model.
And I'm going to be using here pinecone as the vector store.
Lastly, let's go and add a bunch of logs.
Once the pipeline is completed, that will tell.
How many documents did we scrape?
How many URLs did we have?
How many chunks did we index.
And yeah, we want all those stats.
Cool.
Let's go now and run all this code here and let's see what we get.
I also want to open pinecone so we can see everything in real time.
And we can see live the documents being indexed.
So right now we can see we don't have any documents in the vector store.
And let's wait until we get the documentation.
And right now I'm using the code from the optional video.
I'm getting all of it manually.
So we got the documentation.
We severely map interview extract and now we're chunking everything.
After we're chunking, we want to take all of the splits and we want to create batches.
And each batch is going to be indexed in a different call.
So now we can see we indexed a one batch.
Let me refresh the page.
And it might take a couple of seconds until we see the actual vectors in the uh in the vector store.
So let me wait for a second.
Let me go and refresh it one more time.
And right now we can see we have the vectors which are ingested.
So we have the actual content and we have the source and we have the vector ID.
Now it's also important to note that we have also the original text.
And this is important to save because there is no backwards function from a vector to the text that
it represents.
So the embeddings function doesn't have an invert function that correlates to it.
And if I'm going to geek out for a moment and remember my math classes in university, this is a one
way function, the embeddings function.
So it's important to save also the text.
So when we get back the query from the vector stores doing direct.
So doing the retrieval we getting the text back.
So we're not only getting the vector.
So this is something very important to note.
And we can see that we manage to index 6506 documents.
And you can see here in the right side the number of documents that are ingested in pinecone.
So the numbers should match.
And if you don't have the same number so either some of the batches failed or you need to wait a couple
of moments until it finished syncing.
Alrighty, I want to show you something else here.
Now remember when we talked about the retry mean seconds?
So let me go and remove these arguments from our embeddings.
Object we of LinkedIn and let's see what happens.
And I want to show you that if we remove it in this constellation of all of our documents, we're going
to be rate limited.
So you can say removed it.
Now let me run everything again and let me show you the error that we get.
So we're expecting to get this error once we have all the documents.
And we are going to index them.
Because in that process we are going to embed every document and turn it into a vector.
And that's where we're going to hit the rate limit.
And because we don't have the retry mechanism, I mean we do, but we have the default which has a lower
value.
Then we're going to get rate limited.
All right.
So now we are indexing the documents.
So the first step is to turn each vector each document into a vector.
And let's wait and see what happens.
And boom we get here an error so we can see we failed.
We can see the batch number and we can see we get a 4 to 9, which is a rate limit error.
And we can see that the rate limit came from us trying to embed with text embedding three small here.
Now we can see by the way in each error code we can see how much time do we need to keep waiting here.
All right let me go and bring that back.
And now let me use Chrome ADB instead of pinecone.
So I'm going to rename the variable in chrome to be vector store.
And everything should run the same here.
And notice now when we are starting to run it, then we're going to create a chrome adb directory.
And you can see it on the left side here.
And we're going to start indexing there.
And we can see now that Chrome is actually using an SQLite DB.
All right.
So now we can see we're still getting rate limited.
And this is because we haven't really wasted the time we needed to wait in between the and the runs.
So.
And don't mind that, but let me show you that we actually go and index everything in Chrome at this
time and not in pinecone.
All right.
So we can see that here we manage to index a sum of the batches.
Sum failed.
And but anyways everything is saved here in Chrome adb.
If we want we can use it as well.
It's persistent because it's saved in the chrome adb directory.
Alrighty, so I hope you enjoyed the video and you enjoyed the ingestion pipeline.
And the next step is to go and do the retrieval.
So let's go and do that.
And by the way, all the videos that you saw in this section are actually new.
So right now I used a different index name.
My index name in pinecone was linked in docs 2025, and I'm going to have a different index name in
the rest of the videos.
So just giving you a heads up.
Cool.
So see you in the next video.