Okay.
One in this section, I'm going to show you how we build out a loop that puppeteer can go through to
visit all of the different URLs that we get from this first page.
All of the URL properties that we get in our array.
So how do we build out a loop that puppeteer can go through and visit each page so we can scrape out
the job description text for each page.
So first of all, let's rename this function here to be something called scrape.
Listings.
And then instead of having this scrape listings, we are going to have a new main function.
So async function main.
And then in here, let's cut these two browser and page instantiations into the main function instead.
Just like that.
And then from here, we are going to call this and say const listings, await scrape listings.
And in the scrape listings, we're then going to pass in the page variable so the listings function
can use it.
And then also make sure to have it inside here.
So now we created a new main function.
Make sure to also call that down here.
We renamed this function to be scraped listings.
Instead, we pass in the page variable so we can use it inside of this function.
And we instantiate the property browser out here in Main so that we can use it in other functions that
we build out.
For example, this loop where we're going to make now.
So then we end up with this listings, which are listings array, which we also need to remember to
return from this function.
So we say return.
It's called results here.
Let's rename it to be listings.
So it returns the listings array from this function.
Let's call the console log and put it down here.
So we make sure that we are getting the right data from this function call.
So then we have restructured our code a little bit so that we can keep on using the puppeteer browser
in other functions.
Now we need to build out our new function for our loop to go through all of these different URLs.
So let's call it const listings with job descriptions.
And we can say await scrape job descriptions.
So this function needs to take in the current listings URL or current listings.
Arrays of URLs we made before.
So we pass in the listings and then let's also pass in the page because we need to navigate around inside
properties here.
Okay, so with that, let's write out our new function with the loop.
So async function, scrape job descriptions and we have the listings and the page.
Now in here, we need to have this loop for each of the URLs so we can say for var, we start at zero
on the array zero index and we go until we get to the end of this listings array.
So listings, dot length, and then we just go through each of them one by one.
And then we can simply do something similar to what we did just for one page up here.
We can say await.
Page dot go to listings on the I index and take the URL property.
So then we go through each of these listings, job description URLs and visit the page inside properties
here.
And then we can extract the HTML content by saying something like we did up here where we say const
HTML.
I wait await page content.
And just like that, we get the HTML from the page.
Now, the reason some of you might have noticed that I'm using this older kind of for loop instead of
a newer kind of is six for each loop.
So instead of something like this.
For each listing.
And then have the code in here instead of a regular old for loop.
The reason why is because for each sort of dos thing does things concurrently in parallel.
And that doesn't work very well with property.
You need to do things one by one, one page at a time.
It can only visit one page at a time.
You can only sort of do things in serial with property.
It doesn't really handle concurrent requests you make to it.
So basically what I mean is that if you try to do a for each loop with puppets here, it's not going
to work out so well for you.
So that's why I'm using this kind of loop here.
Just so you keep that in mind.
If you're wondering why I'm using this old kind of for loop.
Anyway, now let's try and see if Puppeteer is actually visiting this these sites.
And then in the next section, we're going to actually extract the data.
So let's first try and run it inside Node Index.js and see if it visits all the sites.
So there we go.
I can already see that it is visiting all the different job descriptions and I'm just going to stop
my loop now and make sure when you're running this that you don't run it for a long while because it
is making quite a lot of requests at the same time.
So we need to have some sort of way to limit the rate of requests that we're making so we don't make
too many page visits.
And then Craigslist might ban you from their page because you're making too many requests or putting
too much load on their page.
Now it is kind of unlikely that you're going to do this with puppeteer because it is sort of limited
to how many requests it can make per second.
It is not that many after all.
But in the next section, I'm still going to show you how we can limit the rate of request and make
sure we don't get blocked when we are scraping lots of pages on our website.
So I'll see you in the next section.