Hello, everyone.
In this video I'm going to talk about how to scrape a page that is using pagination.
If that's.
If that's how you pronounce it.
Pagination.
Anyway, what is pagination?
I hope I'm pronouncing the right way throughout this video, which when I'm going to say it anyway,
what is pagination?
So pagination is when a page is dividing itself up to several pages instead of having one long page.
Sometimes they do that where you have infinite scrolling, like in Facebook, Instagram, and so on.
And but here instead, it's divided up into several small pages.
So, for example, here you can see in Craigslist, we can click on next and we go to another page or
click previous and we go to the previous page.
Same thing with eBay, for example.
We can go and search on something.
Like an orange pie and.
Down below in the very bottom.
At the very, very, very bottom.
We have small pages you can go to.
So that is pagination.
And a lot of you guys have been asking me, how do I scrape a page with pagination?
Well, actually, it's not that hard.
It is pretty simple once you know how it works.
So notice how when I click next here, there's an extra like query parameter coming onto it that says
120.
When I click next again, it says 240.
Then I click next again, it says 360, so it's just adding 120 to the page.
And if I set it to zero here, guess what?
I get to the start page again.
And it's the similar inside of eBay.
Here we we I clicked for the next page and we see I get to here where it says page number two and here
it is, setting 200 items to be shown.
So if I set page number three here.
I get to the third page.
So that is how most of the websites pagination is working.
There's a URL and you can change a number in up here to get to another page.
Also, we have a section inside of the Airbnb scraper section where we actually learn how to scrape
Airbnb, which is also using pagination on a certain section of their site we get to.
But here's a really simple example of how to scrape the Craigslist page, because that's a really simple
site doesn't use any JavaScript.
So we're only going to learn and focus on how to scrape a site with pagination.
Now, let's see.
So we get this URL here with the zero at the end.
And I'm going to make a new project here.
Dah dah dah.
I'll call it pagination.
And I go inside pagination and I initialize NPM.
NPM init dash dash.
Yes.
So we can add some packages.
We are going to need request and request promise.
And then.
Cheerio.
And once that is done, we are going to open up the project inside our editor.
So here I have the project running and now I'm going to put my browser to the right of the window or
the screen.
So here I have my project.
I just started up with npm init adding the request request promise and share your packages.
So let's make a file.
Let's call it index, as always.
And let's import the modules or packages we just imported before.
So request say require.
Request promise.
And.
Cheerio.
You say require.
Cheerio.
And I'm just going to close the tab here so you can see more of my screen.
Now, the first thing we need to do is we'll make a scrape function.
A async function scrape.
Because we like to use async await.
It's a very nice clean syntax to use.
And then I call it down here in the global scope.
So we have this URL, this one.
And you can see what we're going to do is we're going to make a for loop and we're going to go through
the first page, which is this one at zero.
Then the next page is 120.
The next page again is 240.
So we make a for loop that starts at zero and adds 120 each time until we get to let's see how many
pages there is.
360.
So.
Let's get to it.
I just cut the URL there, but let's see how it looks with the with the for loop.
So for.
And let's use let instead.
That's modern way to do it.
We start at zero and we continue until index is equal to or mean index is.
More or equal to 360, which is the end page inside of the Craigslist.
And for each time we run this loop, we are going to add 120 to the index.
Now, let's see if I can.
Make it so you can see all the text.
So and then we still have the URL here, so we need the HTML from request.
So we say request get.
And paste into your URL.
Now, obviously I'm going to make the text a little smaller so we can see.
And we need to put in a variable here, which is the number, the index.
So I'm just going to put a plus index here.
So first time we run this for loop, it's going to be zero.
And the next time we run it, it's 120 and so on and so on.
Two 4360.
And.
So that is, remember, we need to have a weight in front of the requests.get.
Because it is an asynchronous call.
And yeah, so we make the Cheerio.
And selector.
We can select inside here from the HTML.
And that is also a weight that's also an asynchronous function.
And now we can select items inside of the HTML.
This is just like the the original Craigslist example we did before, and we're just focusing on how
to do pagination right now.
So let me just see really fast how we can get the title.
So the title we can get by saying.
Um.
Result title.
Get the result.
Title Class.
So let's see.
So for each of the result title classes we go.
We have an index and an element.
With each of these, we can do a console log.
Where we get the element text.
Oops, I forgot.
I forgot to have a.
Parent phases here.
Let me see.
There we go.
So now we get all the titles of all the whatever we are looking for.
I think this is volunteers on Craigslist.
And so that's how we get the titles.
We can just copy this.
And paste it in here.
So result title for each have index element and we console.log the title of the result title class.
And let's also just do a console.log here at page number.
Index.
Okay, so let's see if it works.
There we go at page number one zero.
Page number 120.
Uh, page 240.
Page 360.
And let's see, that's the end of the pages.
So.
If we go to the end of these volunteer jobs.
We should have this art post here of someone looking for someone to clean their home, which we also
see here at the end at page number 360.
So that's how you do pagination, how you scrape a site with pagination.
Of course, Craigslist is a really simple site because it's a static website, but I didn't want to
make this into a complex example.
Basically all sites that have pagination works this way.
There's a certain URL you get and you have to watch how the URL looks as you click on the pages in the
site and you're going to figure out there's some sort of pattern to it.
Some pages only.
Just do a one, two, three, four.
Other pages do something like Craigslist where they have one where they start at zero one, 2240, and
so on.
Or like in the eBay example we saw before.
That's a that's also a classic pagination.
Where we have the page number here at three, and then we can just set it to one to go on page one.
So basically you just make a for loop, usually a for loop and set some numbers and decide how much
you want to increment it by.
And then you just scrape the page for every for loop.
That's how you scrape a page with pagination.
So I hope you could use this for something and let me know if you have any other questions.