Hey there.
Eden here.
And this video is optional.
I wanted to introduce you another powerful way to scrape and extract content using the toolkit.
And like I showed you in the previous video, the main recommendation for most cases is to use the Velcro
because it's super simple.
You just give it a URL and it automatically maps the entire site, scrapes everything in the sitemap,
and even lets you filter exactly what you need using natural language instructions.
So, for example, we can point it to a documentation site and tell it.
Just give me everything about agents and it will do all the filtering and scraping for us.
However, sometimes we want to have even more control.
Maybe we want to customize every step of the process, or we want to go really deep into certain parts
of the website.
So in this video, I'll show you how to use map and to really extract for our use case of extracting
the link chain documentation.
If we want a bit more flexibility and control over the entire scraping process.
In this video, I'll be covering some batch processing strategies, and I'll be going a bit deeper into
scraping with third parties and rate limiting handling.
So feel free.
This is a bit of an advanced video.
And this is again optional.
So we're first going to map the sitemap of the documentation and get all of the URLs that the documentation
has.
And then for each documentation URL we're going to scrape it with the video extract.
Now we're going to run the extraction and scraping concurrently.
So the solution we built here does scale.
And the code I'll be showing here is going to be very similar to the code I showed you in the previous
videos where we covered the map interval extract.
So just a few logistics because it's going to be a pretty long video.
I highly suggest you first watch it and only then try to do it yourself.
And of course, you don't need to write everything from zero.
You can simply copy the snippets that I'll be using.
You can copy it from the repository.
So this is how I recommend you watching this video.
All right.
Let's start by writing some logs before we begin.
So let's log by writing log header.
And we're going to call it with the string documentation ingestion pipeline.
So this is going to be a nice header.
Then we want to log info to map.
Starting to map documentation structure from.
And this is the URL of the LinkedIn URL.
And let me write it in purple by the way feel free to stop and to write it yourself, or to copy paste
the code from the reference code that I'm going to provide in the videos resources with all the code
to this a couple of videos.
All right.
So now we want to invoke the map object.
And I remind you they invoke here is actually using the map tool which is a wrapper around the API to
turn it into a LinkedIn tool.
All right.
So the result of this function execution we're going to get in the variable sitemap.
And the actual list of URLs is going to be in this variable which is going to be a dictionary.
And it's going to be in the key of results like I showed you in the previous video.
So when we log the results of how many URLs we found, we're going to access the results key here.
Cool.
So let me put a breakpoint here and let me run this in debug mode so we can see the objects and the
results.
So now we're going to run it.
And we can see now the header log documentation ingestion pipeline and really map starting.
Let's wait a couple of seconds.
And now we can see the sitemap variable.
Let's open it up.
It has here a results key.
And here we can see a bunch of URLs of the documentation here.
So we can see we can even go even down a bit.
We can see we get here 100 and we can see the rest below.
Let me just go and finish this execution here.
And we can see here the last log that we managed to scrape 500 URLs.
So we want for each URL that we saw, we want to run and execute a request to, to really extract to
get that content of the documentation.
The way we're going to do this is using some batch processing techniques.
So the API also supports batch processing.
So the extract API can receive also a list of URLs in every API call.
So we don't need to make one API call per URL.
We can make one API call for server URLs that we want to extract.
So what we want to do here is after we discover the link chain documentation in all the URLs, we want
to start batching them.
And we want to get from that large list of lots of URLs.
We want to create batches of URLs and each batch is going to be one extract request.
The batches were going to fire up concurrently.
So they're going to execute simultaneously.
So we get here two levels of parallelism that was hard to pronounce.
So the API supports parallel processing in the API layer.
So we don't need to worry about that.
We just need to adhere and not send too many URLs according to their documentation, and that is going
to be controlled by the batch size that we're going to run.
And the second layer of parallel processing is us going to fire up those requests asynchronously.
They are the classic example of I o bound operations waiting for an API calls to complete.
So this technique is going to get us the documentation in no time.
And trust me before doing it, when I downloaded it manually and I didn't do it concurrently, then
it took me a bunch of hours to do it.
All right.
Let me now paste in this function chunk URLs, which is going to receive the list of URLs and a chunk
size.
And it's going to return a list where each element of that list is going to be a list of URLs.
So those lists are going to be the batches of URLs that we're going to be using.
We covered it in the previous video in the notebook.
So I'm not going to elaborate on that.
It's pretty basic Python here.
So now we have the ability to get the batches here.
And let's go now to our main function.
And we want to execute and run the chunk function with the sitemap.
And we want to give it the chunk size of 20.
And we can play around with the chunk size.
Now I remind you don't make it too big because if you make it too big then the API won't receive it
because you'll be sending too much URLs.
And after we're done chunking, we can log it that we chunked it up.
So let me run this and let me show you the logs and the intermediate results that we're getting now.
So let me just put a breakpoint over here.
Let me run this in debug.
And right now we're going to first map the documentation and then we're going to patch it.
We're going to get the patches.
So let's wait and see.
Cool.
So let's check out the patches.
And we can see here a we have here a list where every element in this list is a list itself.
So every list here is going to contain 20 elements which each element is going to be a URL.
So maybe except the last one which is going to be the remainder of the URLs.
All right so let me continue this and we get here 25 batches.
Alrighty.
So now I want to create a function which is called extract batch which is going to receive a batch which
is a list of URLs.
And it's simply going to make one API call for it.
And it's going to add some padding of logging and exception handling.
So let's go and do that.
All right.
So we have this coroutine named Extract batch.
It receives a list of URLs and the batch number.
I forgot to tell you that it also receives the batch number.
And we need this batch number for observability which we'll see in the logs.
We first start by logging that.
We're starting to process the batch number and the number of URLs that it has.
We're then going to await to extract with the a invoke method.
So this is going to run concurrently the extract a functionality to call the extract API.
And the input to this tool is going to be a dictionary.
And this dictionary is going to be um, the format that I really expect.
So they expect to get in their API, the URLs field.
And that value is going to be the list of URLs, which is going to be our batch here.
So we're then going to run it.
We're going to await it because this is a non-blocking operation.
This is io bound.
And after that, if there are no exceptions, we're going to log that we succeeded, and we're going
to log the number of URLs that we managed to extract and to return those URLs.
Cool.
So now that we have this helper function that simply wraps around the extract tool, we want to run
those batches concurrently.
So I'm going to implement a new function which I'm going to call it async extract.
And this coroutine is going to concurrently extract all the URL.
And it's going to do this by creating coroutines for each batch and executing all those things concurrently.
So this is what this a coroutine itself is going to be doing.
So you can see we started by first logging.
And now let's create the coroutines.
So they'll be stored in a variable which is called tasks.
And tasks is going to hold coroutines.
So we want to enumerate over the URL batches.
So to assign for each URL batch we want to assign it with a number.
So we can keep track of.
And we want to create a coroutine that is going to use to really extract.
So you can see over here in the syntax extract batch which is going to receive the batch itself alongside
with the batch number here.
Now if you're not familiar with async execution then this right over here is not really executing yet.
The coroutines because we did not await those expressions here.
So once we have all of them ready so we'll have here a list of coroutines.
Then we are going to gather.
So this is going to await every coroutine in this list over here.
And all those coroutines are going to execute asynchronously.
And we're going to wait over here until all of them finish.
And once all of them finish, we'll have the entire documentation downloaded.
And it's going to be stored in the results variable over here.
And I'll create two variables.
All pages which will collect all successful extracted documents and the failed batches.
So a counter for tracking how many batches have failed.
And we're going to iterate over each element of the results list which is going to be either a dictionary
containing the extracted content in data format.
So it's going to be a dictionary holding the URL key and the raw content, or an exception and an error
object that indicates that the batch has failed.
So if it's going to be an exception, we want to log an error that the batch has failed and to increase
the counter.
And if it's not an instance of exception, then it means that we have a valid result, and for each
element in that batch result, then we're going to have the URL, which is the source of where it came
from.
And we're going to have the raw content.
And from that we want to create a link chain document.
So the link chain document is going to have the page content.
And it's going to have in its metadata field.
We want to give it the source key.
We want to give it the original URL.
So that's how we're going to keep track of which content came from which URL here.
So this is pretty much it.
Now I know this is a bit of confusing right now because you don't see the objects yourselves.
We're going to soon debug this and see.
But right over here in the right side I created an example of what the output might look like.
So feel free to pause for a moment and check out the structure of the results here.
So nothing fancy here is happening.
Simply some data manipulation and data extraction here.
Cool.
So after we're done with the extraction, we want to log the success of the function and log if there
are any failures in our batches.
And we want to return the all pages variable, which is going to collect a list of length chain documents
containing the length chain documentation content.
Alrighty.
So now let's go in our main function and let's add this extraction here.
So I'm going to await async extract.
I'm going to give it the URL batches.
And I'm going to save everything in the all docs variable here which is supposed to be the flattened
list.
I've already put here a breakpoint and run this in debug.
So right now you're seeing the execution.
And it stopped right over here in line 129.
And you can see in the logs, now that we have 25 batches that we're going to execute.
So let me go and continue this and let me show you what we see in the log here.
So it's going to run now the async extract.
Let me go and play this.
All right.
So now we can see we fired up all the batches execution.
And we can see that we're starting to stream the results.
And notice that there isn't any order for the result.
It's first come first go.
So whatever finishes we get that result.
We can see it in the log here.
And let me just paste in the snippets from the next video.
You can ignore it for now, and I simply want to paste it in so we can put a breakpoint and examine
the documents.
So ignore this right now let me just go and rerun everything and let me change the breakpoints to put
it here.
And now let's rerun everything.
So I want to rerun everything in debug.
And I want to show you the intermediate values here.
So right now we are executing the batches.
We're starting to get the results.
And we finished.
And if we check out the all docs here we can see that here we're getting the link chain documents.
So here we have the link chain documents with a the source which is the original URL and the data itself.
So here we see a page not found.
So this means that we might have gotten a wrong URL.
Or maybe this URL does not exist anymore.
And let me check out another document.
And this is an example of the link chain expression language documentation here.
So looks good.
And right now we're ready to continue.