So in the last section, we made sure not to fire all of our requests all at once.
And now in this section, we're going to be avoiding getting banned even further by putting in a sleep
call for each of our requests.
Now, even when we have these sequential requests instead, it can still be quite fast compared to if
a normal user is just clicking around on the Craigslist website.
You see, they wouldn't really be clicking this fast around on jobs.
That's not really a human like behavior, and we want to emulate sort of someone just clicking around
on the website, uh, just slowly clicking from page to page to avoid getting banned from websites and
overloading the web servers.
So how can we do that?
Well, it's quite easy.
We just need to create this, uh, sleep function, which we can call anywhere in our functions when
we want to wait a little bit before we pull the next request.
So our sleep function looks something like this.
We're going to pass in the milliseconds that we want to wait on this function.
And it returns a promise.
And we have this, uh resolve variable here.
And this promise also returns.
Let's make sure we have a new keyword here.
Also returns a set timeout call.
And we call this resolve.
Once the milliseconds has passed on the set timeout it looks like this.
And then we can just call it from anywhere in our code.
Let's try and do also a console log from the index page because that actually fires off really fast
as well.
Let's just do it with the request filed with the index variable here.
And then we can see how fast it fires.
First without the sleep you can see it fires off really fast.
And then if we do an await of sleep with one, one 2nd or 1000 milliseconds await, we can see how fast
it goes now.
And you can see now it waits one second between each request, which is quite a lot, quite slower,
but also much less prone to getting you banned from websites.
So we're also going to call this sleep function down in our descriptions here, just so we don't risk
getting blocked here as well.
So that's it.
Now you have learned how to scrape a static website with pagination, how to get all of the pages and
get all the URLs from these dynamic objects, the job description URLs, and scrape those as well as
well.
And you learned how to make sure your request are being filed in sequential to avoid getting banned
or blocked from websites, and also put a little sleep or waiting time in between each of your requests
to avoid getting banned even further.
So that's it.
Now keep in mind there is some websites that do require JavaScript to actually render the content.
In those cases, we can't use excuse to get just the pure HTML.
We do need to have a full browser running instead, which is something like puppeteer you would use
to render the website and then get the data from it.
I'm going to show you in another section how to use puppeteer to scrape websites as well.
So I'll see you there.
And thank you so much for watching this section.
I hope you got a lot out of it.