One of the reasons I created this mock proxy site is so students of this course can avoid getting stuck
by getting banned on real websites and to provide sort of a safe playground for the students.
So let's try and see how if we do a console log after the request here, let's try and see how fast
the requests are getting fired off.
So if we do a console log just with the job URL here and let's say request fired.
Last job URL.
Let's try and see how fast the requests are being fired off here.
So you can see they get fired off really fast here.
I did also print the job descriptions, but definitely if you did this on a Craigslist website or other
websites, you might risk getting banned because this is way too many, uh, requests to fire off compared
to someone who is just browsing a website clicking around, they wouldn't be making this many requests
on a website.
So the simplest way to get around this is to rewrite our map loop into a good old for loop instead.
So everything is done sequentially, and we can wait for each promise to resolve sequentially.
So that would look something like this instead.
So we do a for.
And we say const job of all.
Chops.
And then we can do simply the same thing here where we say job description page and we get the job description
as well.
Exactly the same, although in this case we will be creating a its empty array instead and then put
the job inside there.
So let's call it all jobs with descriptions.
Let's initialize this empty array.
And then at the end of this for loop here, let's push this, uh, modified job with the description
inside of our array.
So push chop inside here.
And then let's delete all this map loop we have here, because it does fight off all the requests at
once.
At most websites are not going to like this.
So now let's try and see.
Well let's try and do the console log also here.
Let's see.
Request fired.
With the chop, Earl.
Just so you can see how different it looks like when we use this for loop instead.
Let's do the node index.js again.
And it says here all job descriptions is not defined.
It's because I called it all jobs with descriptions instead.
So let's try and run this again.
And now you can see it fires up.
One request goes on to the next one, and it awaits also for the promise to resolve.
So it actually waits for the request to finish and getting the HTML before it moves on to the next one.
So that's slower.
But there is another trick to make it even slower, because imagine if you were someone surfing the
Grayslake website.
This would actually still be quite fast.
I mean, if you're just clicking around on Craigslist browsing things, you wouldn't really go this
fast.
I think this is kind of fast to click around on Craigslist and, uh, look at things.
So you might still risk getting blocked even with this for loop instead.
Now, in the next section, I'm going to show you a trick to avoid even further to get banned from real
websites.