Okay, everyone.
So I think it makes sense that I draw out a little bit about when I want to talk about the disadvantages
and advantages to either the timed method and the on demand method.
So here we go.
We have the so let's say we have the timed method here.
So the time effort, right.
So that method means that let's say we have 1000 users.
So we have 1000 users.
Lots of happy guys here, 1000 guys.
And they want your data.
They want your data that you have on your scraping website.
So my scraping.
Site.
And obviously when you have 1000 users, you're going to get 1000 requests, right?
So each of these lines is a request from a user going to your site requesting this scraped data.
And now, excuse me, this is actually the on demand method we're using.
We're showing here.
So that's the on demand method.
So there's 1000 users going on to your scraping site or your your site exposing an API or just the data
from the site you're scraping.
So that means if you have a scraper behind this API or website, you have a scraper here.
And suddenly this scraper is also going to get 1000 requests.
And at the same time, if you have 1000 users at the same time and this scraper is being run on demand
every time there's a request here.
Well, yeah.
So you're going to have 1000 requests here and this scraper is going to make 1000 requests to whatever
site you're scraping, for example, IMDb, if that's what you're doing, and suddenly your scraper
is going to make 1000 requests as well to this site.
And yeah, obviously that's going to result in a ban just like that.
And so the scraper is going to get banned because you have 1000 requests here.
Right.
And obviously your users are happy because or at least for a while, because they're getting the newest
data from the scraper.
But, um, not for a long while because your scraper here is going to get banned because it's making
1000 requests to this site.
So IMDb or whoever you're scraping is going to go like, Whoa, what is this?
What is going on here?
This guy is making 1000 requests to our site.
Uh, he's going to get banned now.
So that's the on demand method.
This one here where you just scrape as you get a request and obviously you can see that doesn't go well
when you're scaling and when you have lots of users.
But if you want the newest data that's on the site, then okay, it's fine, but you can't scale it
up to having 1000 users.
So that's where the timeout method is a lot better.
So let's see again.
So we have 1000 users, right?
Happy guys again here.
And they make a request to your scraping site.
My scraping site here.
So see lots of requests.
Dada.
Right.
So 1000 requests here from your to your side.
And, um.
Yeah, so.
But then your side is just going to go and look at a database.
So a.
Some kind of database.
And this could be either be this could be MongoDB, this could be a MySQL SQL database, any kind any
kind of database.
You're just returning the.
DV database values basically better your users, but this database is instead being filled from the
scraper.
You're running in a separate location over here.
So you just have a scraper that you run here.
At an interval of
run.
For example, run every hour in one hour.
And.
This site then goes and looks on, For example, the site we're scraping like IMDb.
So suddenly your site is only making one request every hour to IMDb and IMDb is not going to care about
that at all.
They wouldn't care about that.
And then once your scraper has won every hour, it saves the data to the database.
And yeah, so that's how suddenly you can.
Have lots of users and scale up to your scraping sites, whatever you're running.
But obviously the data that we are getting here is one hour old at most.
So your users is not guaranteed to see the newest data.
If you want to use this on demand scraping method instead, you would have to have some kind of limit
here on this layer here and maybe say, Hey, you can only run so and so many scrapers at a time and
obviously it's not the best method to scale anyway.
So for a lot of use cases, this is fine, I think.
Um.
For personal use cases.
Maybe you would prefer just to have the newest data from the website.
So keeping keep that in mind and.
Try to work out what's the best use case you can for your project you're trying to do.
One thing to keep in mind.
Maybe you're a clever now or you think you're clever and think, Okay, well, uh, hey, why don't
we just run the scraper on the website?
I mean, on the client side instead of having it on our API and.
Yeah, that's not going to go so well.
I had one student asking me about this, and basically all new browsers have something called coarse,
which is something to prevent cross-origin, uh, requests from other sites.
So if they go to your scrape, your site, uh, my scraping site, dot com or whatever.
And the client is suddenly making a request to IMDb.
The Chrome browser is not going to allow that.
So client side scraping.
Is not really possible.
It's not really possible because of the course security issue that all new browsers have.
So I would not say that this is a feasible solution.
Um, unless you're, of course your users is willing to download, for example, an extension to Chrome
that disables the core security, uh, which I don't think is going to happen.
Um, one way to perhaps, um, to perhaps avoid this issue.
You think then could be maybe to make a proxy.
So the client thinks that it's making a request to your API, but your API is sending the request further
on to IMDb, but you're going to run into the same issue again, which is that you have um, a ton of
requests from your API onto to IMDb.
You just took out the scraping there and put it on the client side instead.
So, uh, client side scraping and I don't recommend it.
Please don't do it unless you're doing it for personal projects and something like that.
Stick it in the API level instead and try to use the time method.
I think that's the, the best, uh, sort of solution for most of the cases.
Anyway, I hope that you got something out of this and in the next sections we are going to look at
creating an actual API, saving the data to a MongoDB database from our scraper and so on.
So see you in the next sections.