Okay, So today someone asked me, how can you scrape Facebook and and.
Well, Facebook.
After the Cambridge Analytica scandal, they had to revise their API API a lot and a lot of endpoints
were being closed off.
And so I found an article about a guy who's writing about he tried to scrape the desktop version of
Facebook and that ended up with him being banned.
He tried to scrape it using a puppeteer automation and it ended up with him being banned.
So then he tried to disable JavaScript because Facebook obviously has some kind of JavaScript on their
page with which is detecting.
If you're just acting like a human or if you're just scrolling through and getting all of the data.
And so then he tried to disable JavaScript and that ended up with, well, the page Facebook is not
working.
Then he got redirected to mobile dot Facebook.com, which is a more simple design of the Facebook desktop
page.
And well, this is where I kind of disagree with him because he says here it doesn't use any.
However, when you scroll down to the bottom, it updates the page.
So there is some kind of JavaScript going on here.
And also if we.
If you reload the page with the network tab being open, we can see a lot of JavaScript here being loaded.
And if I try to use the extension I have over here, which is disabling JavaScript.
If I try to do that.
Well, the page is not going to load because it needs to have JavaScript enabled.
So I don't know, maybe he had an earlier version that didn't use JavaScript or something, but apparently
they use JavaScript even on mobile dot facebook.com now.
However, however, it is still I think the detection method inside mobile desktop inside mobile version
is probably less effective than their desktop version.
So if you want to scrape Facebook, I would still suggest that you use the mobile variation of it.
Okay.
So however, we are not going to be scraping Facebook in this example because first of all, it would
require you to make a throwaway account for Facebook, which is a kind of a hassle for people.
And second of all, I think it's better that we use this example we have here.
So I found a guy who made a demo page for scrolling or for scraping pages with an infinite scroll.
So infinite scroll means that you scroll down to the bottom of the page and new elements get loaded
automatically.
So you can see in the HTML here.
We have 20 boxes here.
Now, if I scroll down to the bottom, the JavaScript will just add ten more elements.
So that's just some JavaScript inside of the HTML, but it's emulating quite well.
What Facebook and Instagram and other pages like this is doing.
So we're going to go through this example here and try to build a scraper that is scraping this, which
is scrolling down and scraping 100 boxes at a time.
And I'm going to link to the article also both articles, this one where he is showing a how to build
the scraper.
I'm going to build the same scraper just with a little variation, not much, but a little.
And then a I'm also going to link to this article which I am disagreeing with a bit, but you can read
up on it anyway.
Now in the next lecture we are going to start building our scraper of this page and go through it step
by step.
So see you in the next lecture.