So before we ingest the documentation, we first need to download it.
And in order to do so we'll use the map in Tivoli Extract API.
Tivoli map is going to discover and map out the link chain documentation and the website and the URLs
that we want to scrape and extract information from.
The extract is going to extract the data of that pages, which is going to hold all the information
about the link chain documentation.
All right.
Let me go and open this notebook in Google Colab.
So I'm going to use the Google Colab runtime.
And I want now to go and make sure that I have the runtime environment selected in Python three.
Cool.
So we first want to start by installing the dependencies.
So I'm going to click this tab over here which is going to install link chain.
And this is the link chain integration which is highly maintained.
The reason why I am mentioning this is that first of all, it's not obvious that our chain integration
is highly maintained.
And second, if you're using a third party with integration and that third party link chain integration
is not highly maintained, then what may happen is that the third party API might change and it would
break the link chain integration.
So this is very, very important when using link chain third parties, especially when you go to production.
Anyways I'm going to install link chain and I'm going to install certify and certify is simply to make
API calls with a valid certificate which can be verified, which is defensive programming to make us
send lots of requests to the API.
Right.
We're then going to install Rich.
So this is to display some cool text for logging.
And actually we're not going to use pandas.
He got in here because I wrote a bunch of this code.
All right.
So now I want to run the import cell.
So I want to import here a bunch of stuff.
I want to import async IO because we're going to make concurrent requests to extract.
We want to use the OS package in case we want to set up some environment variables.
We want to import SSL because we want to provide a valid SSL context some typing, and we want to import
certify for the certificate.
And here from chain underscore we want to import the extract object in the map object.
And after that we're importing some stuff from a witch.
So this is simply to display a nicely logs.
And then we want to configure the SSL context.
And I don't want to elaborate on this snippet, but at the end of the day it's simply setting up some
environment variables.
So every time we'll be making a request to a third party, which is going to be tavileh, then we're
going to set valid SSL context, and we're not going to get blocked because we don't have a valid SSL
certificate.
All right.
So now we want to go and get the API key.
And if you don't have an API key you can go and click this plus button here.
And just name your API key, create it and copy its value.
So let me copy the value of my API key here.
And you should go and take it and put it right over here.
And then run the cell to set up the environment variable.
And by the way, if you're worrying about my API key then don't.
I'm not exposing it because it's going to be revoked as I'm finishing to film this video.
All right.
So you go and run this cell and initialize your environment variable.
So let's go.
Now initialize an object of map which is going to interact with the map API.
And this API is going to receive an input a URL.
And it's going to traverse the website URL like a graph and to explore all the paths to intelligently
discover and generate a comprehensive sitemap, which is a list of URLs.
So this is the output we're going to receive once we map website.
And we're going to use every element of that list, which is a URL to scrape and extract information
from.
So this is going to be fed into our Rag pipeline.
And I'm going to cover some of the arguments that this API receives.
So it's going to receive Maxdepth which defines how far from the base URL the crawler can explore.
We're going to give it also max breadth, which is the max number of links to follow per level of the
tree.
So per page.
And we're going to give it the limit of 500.
So this means that we don't want to get more than 500 URLs.
Let me go and run this blog.
And let's now continue to invoke this API In the base URL we want to give it is python.com/docs/introduction.
So this is going to be our starting point.
And then we are going to invoke this API and to call it by using the invoke method.
And we can invoke because we imported from link chain.
So this map object is actually a link chain tool like we saw earlier in the videos that can be used
by an agent.
And the way we invoke tools is by using the invoke method or invoke.
And here what actually is happening to really wrapped their SDK with a link chain tool.
So this is what we're actually getting by using the link chain to really package.
Of course we can use it by itself, but I wanted to show you the link chain integration.
So here we're going to invoke this map object.
And we're going to get back the sitemap here.
All right.
So after that we're going to iterate over the URLs that we get.
And we're going to print them nicely.
So let me play the cell and let me show you what I get.
And it's important to note that the results may change depending when are you playing this video.
Because link chain might and probably will add.
Let's change some of their documentation by the time you're seeing this video.
So the important thing to note is that you're going to get with this the up to date sitemap of link
chain documentation.
And you can see now we're displaying the first 50 URLs.
Let me go and click on one random URL.
And let's see what do we get.
And we can see it's a valid documentation page.
All right let's go back to the notebook.
And now we want to select a random page.
And we want to scrape it.
So for that we will Who will initialize the extract object, and I'm not initializing it with any arguments.
And this is going to do the scraping for us.
And it's going to output us a markdown file of the page content.
Let me run this cell.
And now let's go and let's go and extract the content and scrape the content of a certain page.
And let's walk through this code here.
So I want to select some sample URLs.
And here I selected a list that is currently containing one URL, which is the 21st element of this
list.
And you can put here as many as elements that you want.
Then we're going to use the extract a invoke.
We want to use the asynchronous function and we want to go and await it.
And the reason why we're doing it because later when we'll be using it in our ingestion file, we'll
want to extract everything concurrently.
And we're going to give it the input, which is going to be a dictionary with the key of URLs.
And the value here is going to be the list of URLs we want to scrape and extract.
And after we await this coroutine here, we're going to get back a dictionary of results.
And we want to get the results.
Um key here.
So this is where is going to send us the output here.
Cool.
So we get back here a list of documents where each document is going to contain a URL, which is the
original URL that we scraped, and the raw content which is the content of that page.
And then we want to display it nicely.
So this is what's happening in the cell.
So let's go and run it.
And here we can see the scraped content displayed nicely.
Alrighty.
So now I want to show you how to batch, process and extract concurrently many pages.
And this is a must when we want to handle scale, because we want to do everything as efficient as we
can.
And if we have lots of documents to scrape and extract, then we don't want to do it sequentially because
it's going to take forever.
So what we're going to do now is we're going to use the extract API, and we're going to do it in batches,
which are going to run concurrently.
So I have here this chunk URL function.
And this function is going to receive the original list of URLs.
And it's going to receive the chunk size which the default is three.
And it's going to return us a list that contains containing list of URLs.
So we're going to split that big list of URLs.
We're going to split it into a bunch of sublists which are going to be your batches.
Here I have a coroutine, an async function.
Extract batch, which is going to receive a list of URLs.
So this is going to be one batch.
It's going to receive the batch numbers.
So this is for logging.
So we can see what exactly is being processed here.
So we're going to use the extract invoke like we did before.
So this time it's going to work on our batch of URLs.
It's going to return a the result for this and the way we're going to process all the URLs here and
here we're selecting nine URLs to process.
We're going to split it into batches.
So we're going to get three batches of three.
And we're then going to create tasks which are coroutines which are going to run concurrently.
And this is what we're going to do with the async IO gather here.
And we're going to send here the list of tasks where each task is a coroutine which is going to process
a different batch here.
And once this finishes executing we'll get the batch results, and it's going to wait for all the batches
to finish.
And once we do that, we simply go and extract everything into one list.
And that's what we're going to log.
And we want to return eventually.
So this logic over here is the preparation of taking the LinkedIn documentation and downloading it or
scraping it concurrently.
So let me go and run this code block and show you the output.
And we can see that we started executing all the tasks which are coroutines in a simultaneously.
So you can see here the prints are very organized batch one, batch two then batch three.
But the finish rate is different.
We first finish batch two, then we finish batch one.
Then we finish batch three right.
So this is just to show you how everything is running asynchronously.
And if we were to have more batches like we're going to have in the ingestion, which we're going to
do next, we can even see it even more explicitly.