Hey there, Eden here.
And I think now is an excellent time to pause and to reiterate what we did so far and what we have left
to do in the context of our rag ingestion pipeline.
And luckily for us, we are done with the hard part.
And by the way, this is very typical for production grade Rag applications.
A lot of times the hard part is just to get the data to our system.
All right.
So just to elaborate on that.
So far we have taken the source which is the documentation, and we have loaded it into long chain documents.
So this entire process uses external APIs.
It uses to really map to really extract.
We ran everything concurrently.
And there is a lot of depth and a lot of things to optimize over here.
So right now we want to go and prepare everything to be indexed into our vector store.
So this will include two chunk ify and to transform our original documents.
And we're going to split them into smaller chunks.
And we'll soon discuss why.