Okay.
Now, uh, in the previous lecture, we covered how to get some hidden path that was here using a list.
Okay.
And sometimes happen that you don't have a list and you want to get some very useful information from
a website.
Something like this.
Okay.
So I'm using this model, uh, metal de because of, uh, I am allowed to use, uh, my tools on this
and I'm not allowed to use on other website and it's not very good.
Okay.
So here we need to create a program that extracts some useful data from this website.
For example, here you see when I right click here and go to inspect, I see the HTML source code of
this website or web page.
And you see from top to bottom all HTML code.
And in here also we have some very informative data, some useful data.
For example, I searched for each reef.
That is link okay link to other pages.
Here you see we have a link for a CSS link for a css, CSS and also for JavaScript.
If I hit enter some time, let me here.
Now you see that we get a link for index dot php.
And also we have a link for uh login page and a lot of things that we really need them.
And here now I can use a regex and also my program to extract all these links and put them in a file,
or print them into the terminal so I can use them.
Okay.
So let's come here I am going to create a new file Here.
I'm going to name this crawler Chupai.
This time.
And also I need some of this code from here.
So I copy that and then I use Ctrl V to paste them here.
And then I remove this because I don't need to open anything.
And also I need to have this one and also the link.
Okay.
So now I'm trying to access that web page.
So very easy.
We just uh use the response is going to be equal to request okay.
Uh request actually because we have a function called request.
And then in here I'm going to give the URL.
So the URL is already provided right here.
So I give it here.
Now I have the response.
And if I just print the response.
So you will see that I have something like this.
I use two this time hit enter.
I have a response code of 200.
That means I am able to open this.
What if I open?
Want to see the content of this?
So I have content.
I get response dot content.
Now you see that it gives me all the HTML code of this web page.
Perfect.
Here I have all the that thing.
But the most important thing is the link that you see here.
I want to access all the links.
So for that to do that I need to import another module that is called or or regex.
Okay.
So I think in one of the videos We covered regex.
You may know how to use it.
Okay.
So to get only the sheriff or links I'm going to create a variable.
I'm going to name it http links.
It is going to be equal to I'm going to use this okay or E dot find all.
And in here I'm going to provide a regex to get all the links.
So and the first thing that I want is to separate each riff.
And also some of the columns you see we have a strip here and then we have comma.
And also we have something here.
And then we have not comma.
We have this double quote okay.
And also we have double quote here.
So we need to provide something very useful like that.
So.
The first thing is that I want to group these.
Okay.
So I'm going to search for each reef.
And then this sign which is equal and then double quote.
This is one of that thing that I want.
And also after that I want to search for any character or all character.
This is the other group.
And in here I'm going to search for everything okay.
And here I'm going to give that double quote at the end.
So now you see that we have each reef and also this one.
And also we have everything which is inside that.
And then we have that double quote here.
So this will come and search for everything.
But it is greedy okay.
It will include from top to bottom all as one quotation mark.
If we have inside this any quotation mark.
So it is not going to require that okay.
So it is going to move up to the end.
and to remove that we need to use question mark.
And this will remove that this from being greedy.
And also the end like this.
So and this is the point all method.
And also because it is regex I don't need to print the response dot content.
Here I am going to give the response dot content as second argument.
So let's use respond response dot content.
Now instead of printing response dot content, I'm going to print only the links.
Okay.
Use control s.
And also let me come here and re-execute this.
Now you see that we are having a beautiful error here and not use a string pattern on byte like object.
Okay.
Perfect.
We have this one, and you may know that this response dot content is not a string here okay.
It is kind of object.
And we are not able to use the object or object with other thing okay.
So we have this if I just use.
This here is TR and now it is changed to string okay.
And that both now is string.
And string with a string is working properly.
And now you see I only have the A list okay.
A list of all links that exist in that page.
And it is a list okay.
So perfect.
And now I'm going to improve this in the next lecture.