Let's now use the crawl to crawl the link chain documentation and pull out all the documentation.
So web crawling refers to the automated process of browsing website by following hyperlinks, clicking
them, going from one page to the next, and uncovering more and more related content.
And for agents and autonomous agents, crawling is a key capability, especially when we're trying to
access deeper layers of the web that aren't easily reachable through standard search.
Alrighty, let's go to the code and let me show you how easy it is to fetch the documentation.
And this is our boilerplate code right now.
If name equals main then we want to use async IO run.
And then we want to run the main coroutine we have here.
And let's start with adding some logs here.
So I want to log that the recommendation ingestion has started.
And then that we're using to really crawl.
To start crawling.
The documentation in python.com and I remind you log header and log info are in the logger.py file.
Cool.
Let's go and run it just to see the logs are printing and we're all set.
So code is executing.
We're initializing all the clients and let's go and see the logs.
So let's now do the actual crawling.
And to do that we're going to use the crawl object.
And I remind you this is a link chain tool.
Now because we're using link chain Tavileh.
And we want to invoke that.
And we want to provide the URL to be the Python comm.
We provide it with the maxdepth, which is going to define how far from the base URL the crawler can
explore.
And the default is one.
And this has to be an integer and the maximum at the moment is five.
Cool.
So I want to elaborate a bit on maxdepth because this is an important argument when it comes to crawling.
So this is from the best practices for crawling.
And it discusses here the excessive debt problem.
So it's obvious if we're going to have a higher number in Maxdepth then runtime is going to be longer.
This is a no brainer in worst case.
In certain website topologies it can even be exponentially slower.
We usually want to start with depth equals to one or to two, and only then after we review the results
to then increase it if we need more depth.
And this is the smart approach because it's going to give us first faster iterations because runtime
is going to be shorter with lower max depth.
And second, it's also going to be cheaper because we are consuming less of the resources.
So this is the correct approach to start with small.
And we're actually going to do this in the video.
The number five I got after a bunch of iterations and figured out that this should be the number we
want to crawl to get the most documents from the LinkedIn documentation.
And we're going to give extract depth equals to advanced and advanced extraction retrieves more data
including tables and embedded content with higher success rate, but it may increase the latency.
Now, when we invoke the crawler, we're invoking a website traversal tool that explores hundreds of
paths in parallel with built in extraction and intelligent discovery, which we're going to see in action.
So just for quick iterations, let me change Maxdepth to be one right now.
So we're actually going to crawl list links here in the LinkedIn documentation.
And let me run this in debug here, and let me show you the result that we get when invoking this tool.
So we can see now we are starting the crawling.
And we get immediately a result.
And this ran under one second here.
And let me show you the result.
Here we have the base URL where we started.
And we have this results key over here.
And it has a list of all the pages that we crawled.
Each crawl page has the structure of a URL, which is the URL of the page and the raw content.
So let me show you, for example, the page here.
This is the page from the documentation.
And you can see the raw content which is scraped for us.
And I'm super excited about this because crawling is a pain.
It's a tedious task.
It has a lot of room for bugs, tons of things can go wrong and crawling, rate limiting, bot protection,
getting dynamically rendered pages and so many things can go wrong here.
And actually we tried to do this manually in earlier versions of this course and it was really a pain.
So many students had issues and it ran differently on different machines, and sometimes students didn't
get the correct pages.
And it was a real pain.
And that's why I'm super excited of using crawl here.
And just as a side note, as a software engineer, if crawling is not going to be my main business logic,
then I would like to offload it into a third party that knows how to do it much better than me.
So that's in general my approach.
I don't want to waste any time spending, debugging and trying to do this.
All right.
Notice here that we got here 18 results.
Let's go and change Maxdepth to B2.
And let's check out the results that we get.
And let me go and rerun this.
And we got a result.
And we can see.
Let's go and check the results.
Key here.
And we can see that we have here now 75 pages that were scraped.
Pretty cool.
Now let's go and change it to the maximum which is going to be five.
And this is the maximum depth that ability offers at the moment.
Now we can expect that this will be longer because we are going to scrape and crawl much more pages.
And let me fast forward it a bit.
And we got here a result.
Now let's check out what we have here.
It took 26 seconds and we got here 251 results.
Alrighty.
Let me show you something very cool here.
And this is an argument which is called instructions.
And here I'm going to provide natural language which is going to be used by the crawler.
And this is going to specify during the mapping process when the crawler is going to map out the pages,
it's going to tell the crawler which page to scrape and which page not to scrape.
And that's a filtering mechanism that is going to help us get much more accurate and much more precise
results in case we're looking for a specific field.
So for example, here I want to search for content on AI agents.
And let me now run this in debug.
And let me now show you what results we get here.
So let's go now and examine it.
We can see it took 30s I fast forward it.
And we can see now that we have 23 pages that were scraped.
Now, let me show you something cool.
Let me show you the URLs of the pages we scraped.
And just from the slugs of the URL, we can see that it's documentation on AI agents.
We can see that we have information about AI agents over here, and we don't have something which is
not AI agents.
Now, I know we are in a rag section here, but just a quick note that this capability is actually very
useful when we're implementing AI agents.
And one important note on the argument of instructions.
So it really matters which instructions we give tavileh.
If our instructions are going to be bad then we're going to get bad results.
And we need to understand that when Tavileh uses this argument, it actually uses this to help map out
the URLs to crawl or to not crawl.
So it's really a function that helps filter out URLs whether to crawl and extract them.
And that's what we should have in mind here.
That should be the argument.
So we shouldn't put here questions or something like that.
We should put here instruction that is going to help tavileh to decide whether to crawl that page or
not to crawl that page.
And if you're interested in more nuances and how to get the best out of crawl, then I highly recommend
you checking these best practices for crawl page here that I've posted, which has a lot of cool tips
and a lot of cool ways to use this API.
And by the way, remember when we talked earlier about the Maxdepth argument?
So because we're going to use instructions, then we could make the Maxdepth higher because instructions
are going to help the crawler to skip irrelevant pages.
All right.
So let me minimize it.
And let's go back to the code.
And now I want to create from each result I want to create a link chain document.
So we're going to iterate through all the results in the results key here.
And we want to create a document object a link chain document.
And it's going to have the page content which is going to be the result.
But it's going to have raw content fields like we saw earlier in debug and in the metadata.
I want here to put a dictionary with the key of source.
And here I'm going to put the result in the URL key here.
And we're going to use this metadata when we retrieve the information to know exactly where did we get
the source from.
So this is very important.
That helps with user explainability.
To explain the user why our AI rag application answered the way it answered.
And it really helps to create trust in the system.
So this is going to take the information from Tavileh.
And each result we were going to convert and cast into a linked chain document.
So we can later split.
And let me run this in debug.
And by the way, notice I changed the Maxdepth to be one simply for faster iterations because we were
going to be scraping and crawling less pages.
So it's going to be easier in the credits as well, and it's going to improve latency.
So let me show you here the old docs here.
And we can see now we have link chain documents if the page content and with the metadata of our URL.
All right.
So let me tell you what's going to happen in the next couple of videos.
The video after this video I'm going to show you how to crawl the LinkedIn documentation using map and
to really extract, which is taking this process and breaking it down and making it a bit more granular
if we want a bit more control.
This video is optional.
You don't have to watch it.
This is simply to show you if you want more control, what you can do, and it also helps you appreciate
this crawl endpoint.
All right.
So just to recap of what we have so far in the pipeline.
We took the documentation we used to crawl.
And we loaded the LinkedIn documentation into LinkedIn documents.
And now is the text splitting part.
So we want to take each document.
And we want to chunk it up into smaller pieces so we can index them in the vector store.