Hey, everyone.
Today I had a question from a student who is Sean Smith?
He's asking, how can you scrape a table that has multiple pages and uses the same URL?
So a website where the URL doesn't change the page doesn't refresh as you see when you change the page
or change the.
Yeah, change the page inside the table.
How can you scrape a site like that?
Well, Sean, when you say that it doesn't have when it has the same URL and it doesn't refresh, then
that sounds like it is using something called async Json or async JavaScript or as it's called, which
means that we are usually using an API behind in the back end to get the data from.
So the JavaScript is fetching from a backend using usually a rest backend and that is a good thing for
us as web scrapers because that means that we can just get the data in a really computer readable format
in Json format.
So this is the site that Sean mentioned.
And one of the things that I mentioned in the beginning of the course is that you should always check
if there is a back end in the site that you're trying to scrape.
Check if there is a back end first an API.
That's because that's going to make your job really easy.
When you try to get data, it's going to be so easy for you if you can find a rest API to get your data,
it's going to save you from actually building out a web scraper in the first place.
The second one, the second step if it doesn't have an API.
Second step is to use request NodeJS.
The third step if request NodeJS doesn't work.
If the site requires JavaScript but doesn't have a rest API, then you use puppeteer, which is a fully
automated browser.
So rest API NodeJS request pop.
It's here in the in that order.
So based on how difficult it is to build and how prone it is to crashing properties, a lot more prone
to crashing than NodeJS requests and uses more resources and and calling an API is so simple and doesn't
it?
It's so simple.
Anyway, let's go ahead and see how we find the rest API for this site.
It's really simple, so we just go and press F12.
So open up the developer tab in Chrome and I have the network tab selected here so I can see all the
network requests the site is making.
So then we go down here to all the stats on this table we have here.
There's an arrow here to go on to the next page.
So I click on that and now we see it made a request to the rest API.
So they say here football API and what do you see?
What do we see here?
Well, we have lots of data.
We have all the data that we need.
Actually, it's even more than this inside this table.
Sometimes that's an added bonus of finding a rest API is that you get even more data than you can actually
see on the site.
So anyway, so we can see here we have.
Or the wrist or the Json data.
I have this online Json viewer.
I put the data in here, then I can see it in a nice tree if I want to.
We can.
We can look around here.
I can also paste it inside Visual Studio code, save it as a Json file so we can save football Json.
And now when I save it, it's going to format it all automatically because I am using the prettier plugin
inside Visual Studio code and then I can see all the data we have here inside from the Json.
There's even the age.
I think that's the footballers age.
Actually, we have the first last name even formatted as a first and last name.
How much more can you ask for if you're doing web scraping?
I mean, if you had to do web scraping, you had to manipulate the strings and all that stuff.
And yeah, there's just so much here.
Name goals to do, goals, value.
And there's the rank 17, which is actually.
What we only see on this page right here.
Here's this rank 17 and there's the stat, which was also mentioned inside the Json.
So there you go.
That's how we get lots of data from a website when they are using.
If they have an API that you can easily access, sometimes it's not as easy as this and sometimes in
these cases, if the site does require JavaScript and you can't find an API no matter what, then you
have to resort to proper puppeteer.
But if you can get away with not using JavaScript, getting the data, use NodeJS request.
Anyway, that's it for now and I hope you got something out of this video and you can use it for getting
your data from websites.