In the previous lecture.
We extracted these links here and as you see it is not more readable.
Okay.
And also the code that you see here is not very, very user friendly.
And also we need to modify the code that is here.
And also the data that we are extracting, we need to develop or improve the code.
So we could have a better result than this.
So as you see we have a request here.
And also the request is using the Get method here okay.
But we do need to have this function here at all.
Um we have the link here okay.
Now um.
Okay we have this one.
So I need to improve this.
I need to catch this from here.
Okay.
And then add it here.
So remove this function from here.
So right now I'm trying to improve this code in a better way.
Okay.
So we have the URL here.
And let's create another function.
This function is going to be let's name it extract links.
Okay.
So the URL is going to be here.
And inside this method we are going to use something like response that we have it here.
Or let me remove that from here maybe or let that be there right now okay.
We have the response.
It is going to be equal to request.
Okay.
Request dot get It here.
Inside these I'm going to give the URL.
Okay.
Now I have the response here.
And let's return something.
And that thing is re dot find all.
So I remove that from there.
Remove this one as well.
And also in here I'm going to just return the result that is here.
Okay.
We have the response.
We have the content of response.
And we are returning that from here.
That looks perfect.
And now instead of printing a trip that doesn't exist, now we need to do another thing.
So let's create another trip here.
So we have a trip link.
Okay.
This trip links is equal to extract.
Link that we just created.
And I'm going to give the URL as argument here.
So URL.
And now if I just print the URL not retrieved link and save it and then come right here and then hit
enter.
Now you see that still it is working.
And we have the same thing that we were we that we had before.
Now you see again we have the same result.
Okay.
Now instead of just printing the link here, I want to use a for loop to iterate inside them and get
more details.
So.
Let's use a for loop here for link in traveling and in here.
Now if I print the link you will see all this link in each line separately.
So now you see that it looks more bitter okay.
But still you see a lot of thing that we really don't need them.
For example, we have index.php.
We have a lot of things that doesn't necessarily need to be in here, Wikipedia for example.
And a lot of things.
Let me move back here.
For example, I am going to find something like this one.
Not really that one for example.
I am going to find the login page.
Okay so we use login dot php.
So the thing I want is to have this link this part okay.
And then add new thing at the end.
So here you see we have a lot of things.
HTTP index dot PHP.
And these are links that these are not what I want.
For example we have youtube.com.
We don't want that at all.
Okay.
So let's come back here and try to solve this better than this okay.
Uh, to solve these remove some of the link that we don't need.
We need to use, uh, for example, first let me explain something here that you see.
For example, we have index dot PHP.
The page is equal to login dot PHP and you see that it doesn't look like a complete URL.
We need to first handle this one.
Okay.
To handle that we need to import something called URL parse.
Okay so let's use URL.
Uh parse this one and you see that uh, it is it looks like that it is not installed.
So as you see that now, uh, it is not installed.
Looks like it is not installed.
So let's remove this.
And now in Python three, instead of using URL parse, we kind of use an updated version of this.
Okay.
So I'm going to just uh from a URL lib dot parse import something else.
And that is going to be u r l join.
Okay.
Now we can use this URL join to join some of these.
For example you see we have index dot uh this one.
And also in most of these links we don't have https ww at the beginning.
So we need to handle that.
We can handle that very easily right here.
And I'm going to say that link is equal to URL join.
So in here I'm going to join two things.
One of them is going to be your URL.
And I'm going to join this with link okay.
So let's save it and re execute this.
Now you see that uh those thing that we didn't need is removed.
Okay.
And also you see some of the the links is printed here like this or was or something we don't need really.
And I want to I want only these links.
So you see we have OWASp here OWASp, OWASp okay.
And also this is some link like hacker and a lot of things that we really don't need.
For example WW dot php dot net.
I don't need that.
So I need to handle this as well.
So to handle this we can uh use an if statement here.
So and in here I'm going to tell that if the URL that we have is in link, then print that link for
me.
Otherwise not.
So I am going to save it and let's execute this once again.
Now you see that I am having only the links that is, uh, that I need.
Okay.
For example, and these all have, uh, this thing at the beginning.
And also it is what I have.
Okay, what I need.
And if you wanted to do this attack on a real, uh, website, not a real website on a live website,
not HTTP website, not more mortality.
You can give the link of that specific website here, and it will give the appropriate links that you
need here.
Now you can use each of them of each of these links.
For example, uh, this one you see I hit logged in and right now I am in mean, this web page.
And if I go into something else, for example, let's see if we could find something very, very useful.
Okay.
These are indexes installation dot php.
Um.
Anything you want.
Okay.
It will give you.
Show logs.
So you see that we are into the logs here.
And there is another page that you need to find.
So perfect.
Now you see that, uh, we handle this very easily.
Okay.
So thanks for watching.
And I will see you in the next lecture.