Today I got a question about how do you handle a bad network connectivity when you try to run the scraper?
If you're living in a place with a bad Internet for some reason or if the server is a bad server that
sometimes is dropping the connection, then this is what you want to do.
You want to get something called request retry, which is going to act exactly the same as we use request
or request promise.
Except for that it's going to retry the connection every time if the connection is dropped.
So you want to get something called request retry.
So we'll just say yarn add request retry or NPM install request retry, add this package and then try
and just make a request like this.
Just like we used before and call the request retry.
Then I will comment out the one we used before.
Like so.
And now, since request retry by default gives you this object response object which has different properties.
One of them being the body, which is the HTML string, which is the one we actually get here.
Um, it's it means that if you want to use the code like it is now, we would have to write dot body
to make sure you load the HTML string.
But since we, I think it's stupid to edit all of the places we do that we can use a default option
to put in our defaults option for full response to false and that makes it return only the HTML string.
Now let me see where the request retry.
Documentation here if I'm correct.
So full response is by default.
True.
We set it to false and it means we only get the HTML string.
I'm going to link to the GitHub so you can read up on the documentation for this package.
Of course.
Um, but let's just try and see how this goes.
So now we have request as request retry.
And now I have tested this out with something called clumsy, which is going to drop some packages.
And what I would see if I ran it with the normal request that we use before then, sometimes the connection
is going to drop and it will not get the data at all versus using request retry, which means it's going
to retry the connection again.
By default, it's going to retry five times and it has a delay of about five seconds.
So it's going to wait five seconds and then try again or five times.
But if you want to make it, try more.
You can also set max attempts to something else.
You can set it to 110.
I don't know one.
You can also set the delay, retry delay to something longer or shorter, depending on what you find
is the best option.
Now, you could also make this loop into a, um, sort of a single threaded loop, meaning that, uh,
you would only run one request at a time.
You wouldn't run multiple requests at once, depending on how your connection is.
I think that should be your LAStrillionESORT if you want to make that work.
It's going to be really slow if you do it one job at a time instead of doing it, um, multiple jobs.
But if you want to see how you could do it, I have added, uh, the code which is inside of the GitHub
for the Craigslist scraper inside of the branch handling network errors.
In here you can see inside of the index.js file, you can see a method or a function I call scrape description,
not concurrent, which uses a for loop instead.
And the rest of the code is the same.
But since it's a for loop with a await inside, it's going to run just one request at a time.
But like I said before, it's going to be super slow.
So I recommend that only as the last option if request retry doesn't work out for you.
Okay, so that was how to handle bad network connectivity in five minutes.
I hope this helped you a bit and you can get further on with the wonderful world of web scraping.