Okay, so apologies that I went a bit ahead in the last lecture and started writing out the call to
this function.
And but at this we got to see how the extract items function is running and the results it's returning.
So it's returning this node list of all the boxes, right?
And now it's time for us to write the function here to the function for scrape infinite scroll items.
So async function scrape, infinite scroll items and we pass in the page the function extract items,
the target item count, and a delay which we set by default to 1000 milliseconds.
So that's one second, right?
And.
Now let's try and define an array of the items first and then we say a try catch clause.
So catch error.
Console error.
Error.
And.
Then what?
Let me explain first what the basic algorithm here is.
Um, we check how many boxes is in the Dom, and then if it's below 100, then we keep scrolling.
So here we see.
Well, now there's, I think 20 boxes.
So we check.
Okay, that's below 100.
So we keep scrolling.
Then we scroll down more.
Now there's 30 boxes.
Well, that's below 100.
So we keep scrolling and then we get 40 and so on, so on.
You get the idea.
That's the target item count, which we set to 100 here and pass in.
And so let's see, let's see.
We define the items array here and then we do a while items length.
So items is all of the boxes we're getting from this page here.
So while the boxes count is less than item, uh, sorry, target item count, which is set to 100.
Then we need to scroll down and yeah, scroll down on the page some more.
So we see we scrape the items again on the page.
So we say, wait, page dot, evaluate.
Pass in the extract items function the one that gets all the boxes and.
Then we see that.
Let's see.
So then we need to scroll down on the page.
And how do you do that inside of Puppeteer?
Well, Puppeteer doesn't really have a function to scroll down on pages.
What they have instead is what they're pointing you to do instead is to use something called window
scroll to.
And it takes in an argument and a Y argument.
So if you say oh ten, it's going to scroll by ten.
Um, and then we have something called document body scroll height.
So that's the height of the body element, which is basically the whole page.
And if we refresh the page here.
And write document.
Body and scroll hide.
We can see that it is 1495 now.
But if I scroll down a bit, it's going to extend the page and put more boxes on.
So now the height is 2775.
So it's it's longer now and so on.
If you keep scrolling down, we see it gets longer and longer.
So now how do you actually scroll down using JavaScript in the in the page?
The way to do it is we right window scroll to then we have an argument.
We just set it to zero and we pass in the document body scroll height.
So that's just going to scroll to the bottom of the page and that means we are going to get more boxes
loaded into the Dom.
So if I press that, it goes to the bottom of the page and more boxes gets loaded.
You can see here in the elements, we have 40 boxes.
Now, if I pass in this function again, it loads more boxes.
See the scroll bar?
It went up here and in the elements, we now have 50 boxes.
So that's basically what we're going to insert and do inside of the puppeteer commands we have over
here.
So we type in, await page, evaluate and say.
Window.
Scroll to zero.
Document Body Scroll Height.
And then for good measures.
It's a good idea to wait and see if we actually executed this function.
So we say, wait, mean wait for function, and then we pass in um, some function to check if our scroll
height is bigger than our previous height we had.
So we say some backticks here.
Notice that I'm using Backticks here so I can enter in a variable.
So document body scroll height.
Check if it's higher than my previous height.
Previous height.
And.
Well, if that's the case, then we're good to go.
And of course we need to say previous height in a variable.
So let's go ahead and do that.
So let's define previous height up here.
And we say previous height.
Is equal to await page evaluate document body scroll height.
Now, let me zoom out a bit.
And so then the final thing we can have is we can wait for a delay to finish so we can say await page,
wait for scroll delay.
So that's just going to wait for the X number of milliseconds we set up here.
And well, if you're doing this on a live site like Facebook or Instagram, maybe you want to put this
delay to some random range that you set.
So it's not always one second, because if you just set it to one second, always, they might be able
to detect that and hence you get banned anyway.
In the end, after all this, we return the items.
And I also made a mistake here by defining items up here.
I only need to define it outside of the try catch clause up here.
And, um.
Yeah, let's see if it works.
So Node Index.js Now it's going to open the browser and start doing what we're setting it to do.
So let's see.
Let's see.
It looks like it is stuck right now.
Let's see what could be wrong here.
Um.
Yeah, I need.
I missed.
I put in.
I missed a parenthesis.
Here.
Let's try again.
Scroll delay is not defined.
Oh yeah, I need to call it scroll delay up here as well.
Sorry for that.
Okay, So it looks like it is working like it's supposed to.
Now we can see that the items are being increased by one second every time until it gets up to 100,
it should stop.
And then we should have a printout of all of the items.
So just like that, ladies and gentlemen, that's how we can keep on scrolling on a page and scrape
all of the elements.
So keep in mind the errors I made.
I was missing a parenthesis here and I needed to call this scroll delay instead of delay.
And with that being said, that is how you build a scraper for a infinite scroll type of application.