Hello, everyone.
In this section we're going to look at how to deploy a scraper to Google Cloud platform and how to run
it at a specific interval.
So again, I have the same scraper that we introduced from a previous section, except for this time
I have added a little rest API.
So this means that we are going to run this scraper by doing a get request on just a base URL here.
So if I go ahead inside of Postman and I just run a get request.
On the base URL.
It's going to return course script and it has saved the cars or the Craigslist cars into the MongoDB
database right here, just like in our Heroku app, except for in our Heroku app.
We just had a NodeJS application and Heroku set up a cron job to run this NodeJS application in a specific
interval.
The way it works inside of Google Cloud platform is that you set up a cron dot yaml file where you specify
a specific URL that you want this cron job to do a get request on and then you specify a schedule for
it.
And now this cron job then is going to do a get request on a URL, which is just the base URL here.
And then I set up a get endpoint on this base URL using Express Rest API.
And I simply initiate the main scraping function in here.
So it's exactly the same as our previous section or the original scraper I created and deployed in Heroku.
But now we have just a get endpoint where we initiate the scraper instead.
And that is because that is the only way that you can do a schedule cron job inside of Google Cloud
Console or Google Cloud Server is by setting up this cron job.
And the cron job will then do a get request on a specific URL.
You can't make the cron job, just run a specific command.
You have to set it up to a URL instead.
Now in order to deploy it, it is quite easy.
You just go ahead into the console.
And first of all, you can do the Google app deploy.
And to deploy your app or Gcloud app deploy to deploy your NodeJS app.
And then you do a another way, you add the chrome Yaml file and that is going to run this and then
you will see that it's running a type of cron jobs up here and then it will give you a URL where you
can visit the task or the cron jobs.
And then in the side this URL here you can see the cron job you have set up with the frequency and when
it was last run and you can also set it to run now if you want to do that.
So that is how you deploy a periodic scraper into Google app engine.
And I hope you got something out of this and you can use it.
One more thing I forgot to say is that you can check out the code I made, of course, on the Google
deploy Deploy branch inside of this car scraper code.
So go ahead and clone the repository and switch over to the Google deploy branch and you can see all
of the code.