Hey there.
Eat it here.
So the first part of our brag pipeline is to ingest the documentation.
We're going to get the latest up to date latest documentation.
And we're going to ingest it and index it into our vector store.
And in order to do this we're going to be using the Wacom, which is going to make this tedious task
of downloading the documentation since and before we dive in, I just want to give a bit of history.
When I made this course in early 2022, I was doing this process manually, writing a lot of scripts,
and it really was cumbersome.
It broke.
It has a lot of issues, and right now in 2025, things have changed a lot.
And we have tools like crawl, which are going to help us to achieve this with just a bunch of API calls
just to double click.
So I used to use File Crawl in order to achieve this tracking capability.
However, I found that the Firepro package was not very well maintained with link chain ecosystem and
it had some scaling issues.
So that's the reason I chose to migrate from crawl to the With the Velcro.
All right.
So let's talk now about what we're going to be doing in the next couple of videos.
So first I want you to get to know the API better.
And we're going to have a walkthrough of how to use tabular map tabular extract to really crop.
And after that we're going to be integrated into our chain Rag ingestion pipeline which is going to
take the latent documentation.
It's going to map all the URLs.
And then it's going to concurrently scrape those URLs.
And we're going to add some metadata and index everything into a vector store.
So this entire process it might look like a lot but it's actually not a lot of code.
And all the heavy lifting is going to be done by link chain and the village.
The village is going to be responsible for the crawling and scraping of the documentation, and link
chain is going to be responsible to chunk everything up, update the metadata and to index everything
seamlessly.
By the way, in case you're worried about pricing.
So don't be because the really offers us a very generous free tier, which is more than enough for this
course and for this task.