All right, So now we got our script changed a bit so we can scrape all of the, uh, well, the URLs
and we get the nice array back.
But now we also need to actually go through each of these description pages.
So let's try and do that in this lecture.
So let's create a new function.
We call it scrape description pages or scrape description page.
We pass in a URL
and
do a try catch clause.
It's always nice to do.
Um, let's see.
So we pass in the homes to the description page.
And then what we want to do is we want to go to a new page in this function.
I don't want to launch a new browser for every description URL we are going to go into for every home.
That would be a huge waste of resources to open a new browser or totally new browser window every time
we have a new URL.
Instead, what we want to do is we want to go to a new page that we're just scraping the description
page on and we want to go to the new URLs on this page.
Let me show you what we mean exactly in code.
So I'm going to declare the browser variable up here in the global scope and I'm going to move this
statement to our main function.
Instead, remember to remove the const, so it uses the global variable and then instead of making a
new page for every description page we go through, like instead of making ten new pages, I just want
to use one page for all of these URLs.
So I'm also going to create a new page here.
Description page.
And let's say await browser new page.
So that's going to create a new page which we can pass into our function here.
So add another parameter up to the function call here, which we call page.
And let's see.
Remember to put on weight here, and then we put in the description page.
All right.
Now we just need to make the call in here just to see if it is working.
So we go and say await page, dot, go to.
So that should go through the array.
We get the array of homes and we should go through each of these homes, arrays or URLs and open the
description page.
And so we need to make a for loop.
I'm going to make a good old fashioned follow because a follow up is run in serial, unlike something
like for each which which is going to run it in parallel.
And we don't want to run this in parallel.
It's, uh, it's not going to end so well for you.
I believe it's, uh, you might want to mess around with it later if you want to optimize for things
and run it in parallel.
But for now, we just keep it in serial.
Um, so we get the homes array and we go through each of these arrays and we pass in the URL.
So Holmes I and the description page which we make up here and.
Yeah.
So this should just open each of these description pages in a single page.
Let's see how it's going to look like.
So here it is.
Opening the homes array mean the homes page.
And there we are, opening the description page and it should be going to a new page every once in a
while.
So you can see it's going to a new page and another one and so on.
You get the picture right.
Um, so of course the next step we have to do now is we want to scrape some values from each of these
pages.
And yeah, that's, that's the fun part.